HELP running large code.. having storage issues

I am attempting to sort through csv data points in order to filter a certain data type out to put into a new csv. The large csv contains 55 columns and approximately 43 million rows. I am running into storage issues from Matlab and my computer, even though I have a 500GB adidtional storage drive that I am working through. It takes about 4 1/2 hours to run this code, but I always seem to have an error when I check back. It is not a java heap error, but just states "out of memory. Type HELP MEMORY" and I get a popup box that says low on storage and shuts matlab down. This is haulting my project and I am not sure where to go from here. Any recommendations are appreciated...
clc;clear;
myfile = fopen('file_2009.csv');
whichSite = input('Which site? ');
year = 2009;
recs{10e4,1} = [];
ctr = 1;
while ~feof(myfile)
linestring = fgetl(myfile);
supercell=strsplit(linestring, ',','CollapseDelimiters',false);
if str2double(supercell{3}(2:5)) == whichSite
recs{ctr,1} = supercell;
ctr = ctr+1;
end
end
fclose(myfile);
mycsvfile = sprintf('Site%d_%d.csv',whichSite, year);
recstable = cell2table(recs);
writetable(recstable,mycsvfile);
recs = [];
recstable = [];

1 Kommentar

Adam
Adam am 20 Feb. 2019
Surely you can just read and write your data in sensible-sized chunks instead of trying to store 10^4 cells in memory at once? It seems to just be a sequential operation so I would assume you can just append results to those previously saved.

Melden Sie sich an, um zu kommentieren.

Antworten (0)

Kategorien

Mehr zu Data Import and Analysis finden Sie in Hilfe-Center und File Exchange

Produkte

Gefragt:

am 20 Feb. 2019

Kommentiert:

am 20 Feb. 2019

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by