How to split huge .csv files to multiple .csv files based on size?
13 Ansichten (letzte 30 Tage)
Ältere Kommentare anzeigen
Anjan
am 20 Nov. 2018
Bearbeitet: Walter Roberson
am 13 Feb. 2021
I am trying to split huge .csv files (11 GB) that has both combination of text and numbers into mutiple files based on size (0.5 GB each). I tried using some of the answers in the matlab community but no luck
I hope someone can help!
4 Kommentare
Walter Roberson
am 21 Nov. 2018
Bearbeitet: Walter Roberson
am 13 Feb. 2021
Which OS? The easiest way to do this is with the unix split command , quite easy . For Windows I would look at https://www.gdgsoft.com/gsplit/
Akzeptierte Antwort
dpb
am 21 Nov. 2018
fidR=fopen('originalfile.csv','r'); % open the big file to read
NFiles=20; % number of files to create
NPerFile=round(Nrows/NFiles)+1; % rough number records / file
for i=1:NFiles
try
fidW=fopen(num2str(i,'fileNew%00d.csv'),'w'); % open a file to write
for j=1:NPerFile
fwrite(fidW,fgets(fidR)); % transcribe lines verbatim
end
fidW=fclose(fidW); % close that one
catch
fidW=fclose(fidW); % close that one
fidR=fclose(fidR);
end
is a poor-man's split tossed off at the console here; the rounding for number of records per file should catch the whole file; the error routine should only occur when the last file runs into feof on the last one if my logic is right.
And, yes, unfortunately, we can't always make others do sensible things about how they collect data we're subsequently given, granted...I recognized that was likely the case, hence the smiley.
4 Kommentare
Tanumaya Bhowmik
am 13 Feb. 2021
How will this script work on a .csv file with header? Would it be better to remove the header from the big file, do the split, and then add headers on the top line of each of those split files? How can I concatanate headers as the first line of an entire directory of .csv files? Thank
Walter Roberson
am 13 Feb. 2021
compose() or the undocumented sprintfc() are good at creating separate outputs.
Weitere Antworten (0)
Siehe auch
Kategorien
Mehr zu Environment and Settings finden Sie in Help Center und File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!