MATLAB Answers

0

Matlab uses much more memory reading a file

Asked by Changyue Song on 24 Apr 2017
Latest activity Commented on by Walter Roberson
on 26 Apr 2017
Hi, I have a csv file with 22 columns and 871,000 rows. Columns are separated by comma. Each value is a text or number embraced by quote. The size of the file is about 150 MB. But after I read the file into MATLAB using textscan, the variable which stores the data takes up about 2GB of memory! For another csv file with 47 columns and 7,000,000 rows which have similar structure and is about 2GB in hard disk, MATLAB takes forever to read it using textscan. However, R is able to read these two files and the memory used is approximately the same as the space needed in hard disk. Are there any explanations for this? Thanks.
PS: Thank you for your help guys. Yes I can process the files line by line and discard the columns that I do not need. I am just curious why Matlab uses so much more memory than R or than the original file in the hard disk.

  8 Comments

If some of the columns are known to be numeric, then it would be better to use a more specific format.
You write "Each value is a text or number embraced by quote." Given the quotes, there is no alternative to your format specifier - afaik.
Recently Matlab has become better to handle large files and strings. Which release do you use?
Are the sizes of the different strings approximately equal?
If there are fields known to be numeric but enclose by quotes anyhow, then you can code a literal " character in the textscan format.

Sign in to comment.

1 Answer

Answer by AstroGuy1984 on 25 Apr 2017

Do you need to read the whole matrix at once or are you just doing a routine line by line? If you are just running something on each line, I'd suggest NOT reading the whole file at once and doing something like:
fid = fopen('file.dat');
tline = fgetl(fid);
while ischar(tline)
% STUFF YOU WANT TO DO
tline = fgetl(fid);
end
fclose(fid); %DONT FOGET THIS!
Unfortunately, handling huge data files you eventually have to get creative in how you handle things. I would also suggest looking into using properties such as fseek and frewind if you're needing to pull out specific entries from a huge database. It will save you a ton of time and effort if you're just referencing the textfile for something.

  0 Comments

Sign in to comment.