importing large 4GB+ tab-delimited ascii file makes MATLAB crash
3 Ansichten (letzte 30 Tage)
Ältere Kommentare anzeigen
Sean Phillips
am 17 Mai 2019
Beantwortet: Sean Phillips
am 18 Mai 2019
i didnt have a problem until now the data files have been scaled up in complexity
i usually use
IMPORT = importdata('components.dat')
in this case components.dat is 4 GB+
it is a table of numerical values consisting of 12 columns and millions of rows
for smaller models this has worked but now MATLAB crashes when i try to import at this size.
I have 96GB of RAM and dual Xeons.
Do i need to change setting in MATLAB or is there better ways to import this data
this is an exmaple of the file
1 0.01 -50.00 0.00 -3950.00 -2.0000E+04 0.000 -4000. -1.409E-15 -2.138E-15 0.00E+00 0.00E+00
1 0.01 -50.00 0.00 -3950.00 -1.9900E+04 0.000 -4000. -1.410E-15 -2.193E-15 0.00E+00 0.00E+00
1 0.01 -50.00 0.00 -3950.00 -1.9800E+04 0.000 -4000. -1.411E-15 -2.249E-15 0.00E+00 0.00E+00
1 0.01 -50.00 0.00 -3950.00 -1.9700E+04 0.000 -4000. -1.412E-15 -2.306E-15 0.00E+00 0.00E+00
1 0.01 -50.00 0.00 -3950.00 -1.9600E+04 0.000 -4000. -1.411E-15 -2.364E-15 0.00E+00 0.00E+00
1 0.01 -50.00 0.00 -3950.00 -1.9500E+04 0.000 -4000. -1.411E-15 -2.424E-15 0.00E+00 0.00E+00
1 0.01 -50.00 0.00 -3950.00 -1.9400E+04 0.000 -4000. -1.409E-15 -2.485E-15 0.00E+00 0.00E+00
1 0.01 -50.00 0.00 -3950.00 -1.9300E+04 0.000 -4000. -1.407E-15 -2.547E-15 0.00E+00 0.00E+00
1 0.01 -50.00 0.00 -3950.00 -1.9200E+04 0.000 -4000. -1.405E-15 -2.611E-15 0.00E+00 0.00E+00
1 0.01 -50.00 0.00 -3950.00 -1.9100E+04 0.000 -4000. -1.401E-15 -2.676E-15 0.00E+00 0.00E+00
1 0.01 -50.00 0.00 -3950.00 -1.9000E+04 0.000 -4000. -1.397E-15 -2.742E-15 0.00E+00 0.00E+00
1 0.01 -50.00 0.00 -3950.00 -1.8900E+04 0.000 -4000. -1.392E-15 -2.810E-15 0.00E+00 0.00E+00
1 0.01 -50.00 0.00 -3950.00 -1.8800E+04 0.000 -4000. -1.386E-15 -2.879E-15 0.00E+00 0.00E+00
it is tab delimited
0 Kommentare
Akzeptierte Antwort
Weitere Antworten (2)
Steven Lord
am 17 Mai 2019
Consider making a tall array or perhaps using memmapfile. These tools are linked from this documentation page.
1 Kommentar
per isakson
am 18 Mai 2019
Bearbeitet: per isakson
am 18 Mai 2019
That's strange. I use Win10 and R2018b (64bit). btw: What do you mean by "[...] up in complexity" ?
Anyhow try
>> num = cssm( 'components.dat' );
>> whos num
Name Size Bytes Class Attributes
num 13x12 1248 double
where
function num = cssm( ffs )
fid = fopen( ffs );
cac = textscan( fid, '%f%f%f%f%f%f%f%f%f%f%f%f' ...
, 'Delimiter','\t', 'CollectOutput',true );
fclose( fid );
num = cac{1};
end
If that doesn't work, I guess there is some strange stuff in the middle of your text file.
I added
, 'ReturnOnError',false
to the textscan call and the string, "an error", to the text file an run
>> tic, num = cssm( 'cssm.txt' );toc
Error using textscan
Mismatch between file and format character vector.
Trouble reading 'Numeric' field from file (row number 1005147, field number 1) ==> an error\n
Error in cssm (line 4)
cac = textscan( fid, '%f%f%f%f%f%f%f%f%f%f%f%f' ...
>>
textscan() found my string "an error" on the row 1005147.
0 Kommentare
Siehe auch
Kategorien
Mehr zu Text Data Preparation finden Sie in Help Center und File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!