Filter löschen
Filter löschen

Writing files to .csv

5 Ansichten (letzte 30 Tage)
Angshuman Podder
Angshuman Podder am 8 Apr. 2020
Kommentiert: Angshuman Podder am 10 Apr. 2020
Hi, I am curious to know about the execution speed of a task I want to perform. I have a lot of data (variables) to store that is generated at every time step. Currently, I am dumping the output to a .csv file at each timestep. Something like:
%demo
fid1='C:\Users\Desktop\data\demo.csv';
v1 = fopen(fid1,'w');
for i =1:10000
%say these are the variables that I need to store
x2 = rand(5,2);
x3 = rand(6,3);
fprintf(v1,'%d \t', i);
fprintf(v1,'%10.3e \t', i,x2(:,1)',x3(:,1)');
fprintf(v1,'%d \t', i);
fprintf(v1,'%10.3e \t', i,x2(:,1)',x3(:,1)');
fprintf(v1,'\n');
end
fclose(v1);
Is this the fastest way? Or should I allow matlab to store all the values for all the variables and then after the end of process runtime, I dump them to a .csv? Are either of the process going to impact the execution speed of the actual model code which generates the data?
  5 Kommentare
Walter Roberson
Walter Roberson am 10 Apr. 2020
Suppose you are logging data from an hour long experiment. Suppose that if you lost 5 minutes of data, that you could still get reasonable results. But suppose losing the entire hour would be expensive. In such a case, every (say) 4 minutes you could take the data you had accumulated in the last 4 minutes and try to write it out, and if the write failed you would still have most of a minute to raise an alarm so that the people running the experiment could take appropriate action. Likewise if you were writing every 4 minutes, then if the power failed or the system rebooted for some reason, you would have only lost at most 4-and-a-hair worth of data.
Collecting all of the data first and then writing it all out at once is the most efficient from the point of view of time and system resources used, but that kind of efficiency is not always the most important factor.
At the same time, if you do not absolutely need every value to be logged every time, then it is not advisable to log every result immediately, because that is costly in system resources.
Typically the most efficient hard disk useage is if you write out multiples of 8 kilobytes at a time, with the operating system overhead being least if you write out a number of those at a time, such as a megabyte.
Angshuman Podder
Angshuman Podder am 10 Apr. 2020
This is such a nice explanation. Thank you.

Melden Sie sich an, um zu kommentieren.

Antworten (0)

Kategorien

Mehr zu Standard File Formats finden Sie in Help Center und File Exchange

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by