Out of memory error for big data

3 vues (au cours des 30 derniers jours)
Sharif Khalil
Sharif Khalil le 29 Déc 2020
Modifié(e) : Sharif Khalil le 31 Déc 2020
The masterDataMatrxOffset, contains 464780 rows and 256 columns of complex numbers. I use meshgrid to get 256X256 matrix of the real and imaginary and store it in a cell, but I get this error: Is there a way to overcome the out of memory?
Out of memory.
Error in meshgrid (line 62)
yy = repmat(ycol,size(xrow));
Error in auto_encoder (line 34)
[Po,Ph] = meshgrid(power(obsrv,:),phase(obsrv,:));
More information
tic;
clear; clc; close all;
%% Load Files
load masterDataMatrxOffset;
%% Extract Parameters
iAntenna = size(masterData,2) - 16;
data = masterData(:,1:iAntenna);
power = normalize(real(data),'range');
phase = normalize(imag(data),'range');
Region = masterData(:,261);
iReceiver = length(data);
iRegion = max(Region);
%% Initializations
xTrn = cell(1,max(Region));
xTst = cell(1,max(Region));
datamat = cell(1,size(data,1));
%% Split into training and testing
for idx = 1:iRegion
IDX = find(Region==idx);
train = sort(randperm(size(IDX,1),ceil(0.8*size(IDX,1))));
trnidx = IDX(train);
tstidx = setdiff(IDX,trnidx);
xTrn{idx} = trnidx;
xTst{idx} = tstidx;
end
xTrnidx = cell2mat(xTrn');
xTstidx = cell2mat(xTst');
Trainreg = zeros(numel(Region), max(Region));
Trainreg(sub2ind(size(Trainreg), 1:numel(Region), Region')) = 1;
tTrain = Trainreg';
for obsrv = 1:size(power,1)
sprintf('observation = %d',obsrv);
[Po,Ph] = meshgrid(power(obsrv,:),phase(obsrv,:));
datamat{obsrv} = Po.^2 + Ph.^2;
end
toc;
  4 commentaires
Ive J
Ive J le 29 Déc 2020
Modifié(e) : Ive J le 29 Déc 2020
You still can store them in a cell array
myCell = {};
myCell{1} = tall(1:100); myCell{2} = tall(101:200);
myCell{3} = myCell{1}.^myCell{2};
myCell =
1×3 cell array
{1×100 tall} {1×100 tall} {1×100 tall}
Sharif Khalil
Sharif Khalil le 31 Déc 2020
Modifié(e) : Sharif Khalil le 31 Déc 2020
It still shows out of memory. The cell is 1X464780, but the matrix inside each cell is 256X256.
for obsrv = 1:size(power,1)
[Po,Ph] = meshgrid(power(obsrv,:),phase(obsrv,:));
Talldata = tall(Po.^2 + Ph.^2);
datamat{obsrv} = Talldata;
end

Connectez-vous pour commenter.

Réponses (0)

Catégories

En savoir plus sur Large Files and Big Data dans Help Center et File Exchange

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by