Efficient way to import large date sets
1 Ansicht (letzte 30 Tage)
Ältere Kommentare anzeigen
Hi All,
For the past couple of month I have been working with Intraday quotes for stocks and futures. I was focusing more on structing the data and running the codes for a given day, so uploading the data wasn't my main concern. However I finally got past it and now the process of putting the data into Matlab is taking forever. It's taking me half an hour to upload the data sets which contains 10 columns of numbers, 2 columns of strings (date vectors) and 1.7 million rows on average.
I have no clue of how long it is suppouse to take to get the data set uploaded. If is it too much how can I speed it up?
I have tried the basic txt import and ODBC connection to an Access database and both seemed really slow.
Can someone help me out?
Thanks
Túlio
4 Kommentare
per isakson
am 11 Dez. 2013
Bearbeitet: per isakson
am 11 Dez. 2013
I think you can read your file in less that 20 seconds with textscan. Could you post a few lines of the file?
Akzeptierte Antwort
per isakson
am 11 Dez. 2013
Bearbeitet: per isakson
am 11 Dez. 2013
Try
len = 1e5;
str = '2013-12-10, 00:01:02, 1,2,3,4,5,6,7,8,9,0';
fid = fopen( 'c:\m\cssm\data_2.txt', 'w' );
for jj = 1 : len
fprintf( fid, '%s\n', str );
end
fclose( fid );
tic
fid = fopen( 'c:\m\cssm\data_2.txt', 'r' );
cac = textscan( fid, '%s%s%f%f%f%f%f%f%f%f%f%f' ...
, 'CollectOutput',true, 'Delimiter', ',' );
fclose( fid );
toc
tic
buf = cell2mat( cac{1} );
sdn = datenum( buf, 'yyyy-mm-ddHH:MM:SS' );
toc
I get
Elapsed time is 0.782288 seconds.
Elapsed time is 1.648444 seconds.
If you have 1GB extra ram for the data, it will scale nicely.
0 Kommentare
Weitere Antworten (0)
Siehe auch
Kategorien
Mehr zu Database Toolbox finden Sie in Help Center und File Exchange
Produkte
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!