Out of memory error for big data

3 Ansichten (letzte 30 Tage)
Sharif Khalil
Sharif Khalil am 29 Dez. 2020
Bearbeitet: Sharif Khalil am 31 Dez. 2020
The masterDataMatrxOffset, contains 464780 rows and 256 columns of complex numbers. I use meshgrid to get 256X256 matrix of the real and imaginary and store it in a cell, but I get this error: Is there a way to overcome the out of memory?
Out of memory.
Error in meshgrid (line 62)
yy = repmat(ycol,size(xrow));
Error in auto_encoder (line 34)
[Po,Ph] = meshgrid(power(obsrv,:),phase(obsrv,:));
More information
tic;
clear; clc; close all;
%% Load Files
load masterDataMatrxOffset;
%% Extract Parameters
iAntenna = size(masterData,2) - 16;
data = masterData(:,1:iAntenna);
power = normalize(real(data),'range');
phase = normalize(imag(data),'range');
Region = masterData(:,261);
iReceiver = length(data);
iRegion = max(Region);
%% Initializations
xTrn = cell(1,max(Region));
xTst = cell(1,max(Region));
datamat = cell(1,size(data,1));
%% Split into training and testing
for idx = 1:iRegion
IDX = find(Region==idx);
train = sort(randperm(size(IDX,1),ceil(0.8*size(IDX,1))));
trnidx = IDX(train);
tstidx = setdiff(IDX,trnidx);
xTrn{idx} = trnidx;
xTst{idx} = tstidx;
end
xTrnidx = cell2mat(xTrn');
xTstidx = cell2mat(xTst');
Trainreg = zeros(numel(Region), max(Region));
Trainreg(sub2ind(size(Trainreg), 1:numel(Region), Region')) = 1;
tTrain = Trainreg';
for obsrv = 1:size(power,1)
sprintf('observation = %d',obsrv);
[Po,Ph] = meshgrid(power(obsrv,:),phase(obsrv,:));
datamat{obsrv} = Po.^2 + Ph.^2;
end
toc;
  4 Kommentare
Ive J
Ive J am 29 Dez. 2020
Bearbeitet: Ive J am 29 Dez. 2020
You still can store them in a cell array
myCell = {};
myCell{1} = tall(1:100); myCell{2} = tall(101:200);
myCell{3} = myCell{1}.^myCell{2};
myCell =
1×3 cell array
{1×100 tall} {1×100 tall} {1×100 tall}
Sharif Khalil
Sharif Khalil am 31 Dez. 2020
Bearbeitet: Sharif Khalil am 31 Dez. 2020
It still shows out of memory. The cell is 1X464780, but the matrix inside each cell is 256X256.
for obsrv = 1:size(power,1)
[Po,Ph] = meshgrid(power(obsrv,:),phase(obsrv,:));
Talldata = tall(Po.^2 + Ph.^2);
datamat{obsrv} = Talldata;
end

Melden Sie sich an, um zu kommentieren.

Antworten (0)

Kategorien

Mehr zu Large Files and Big Data finden Sie in Help Center und File Exchange

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by