Write and read numeric data line by line

Here is a working example of what I have in mind:

N=10000; M=10000;
x=rand(1,M);
dlmwrite('file',x,'delimiter',' ')
for k=1:N-1
    x=rand(1,M);
   %In the real application x is the result of some calculations
    dlmwrite('file',x,'delimiter',' ','-append')
end
toc

tic
for k=1:N
    u=dlmread('file',' ',[k-1 0 k-1 M-1]);
    %calculations using vector u
end
toc

The writing part is fine (about 5 seconds on my PC), but the reading is

awfully slow - about 43 seconds. My guess is that it would help if reading

is resumed automatically after the last line read. The command "fgetl"

appears to do this, but is designed for character, rather than numeric,

variables.

Any other suggestions?

(The problem entails generating an array

of size 10^5 X 10^5, which is used by another program.)

Answer 1

[Michael Soskind]

Hi Chris,

If I were you, I would try and use fprintf and fscanf rather than dlmwrite and dlmread.

These are quite a bit faster in my experience, and testing with smaller example arrays, give a much better result.

I compare the two in the following code:

%% Using fprintf and fscanf, which seem order of magnitude faster

N=100; M=100;
x=rand(1,M);
dlmwrite('file',x,'delimiter',' ')
tic
fid = fopen('file.txt', 'a');
for k=1:N-1
    x=rand(1,M);
   %In the real application x is the result of some calculations
    fprintf(fid,repmat('%f ', 1,M), x);
end
fclose(fid);
toc

tic
fid = fopen('file.txt', 'r');
for k=1:N
    u=fscanf(fid,repmat('%f ', 1,M),[1 M]);
    %calculations using vector u
end
fclose(fid);
toc

%% Using dlmwrite and dlmread

N=100; M=100;
x=rand(1,M);
dlmwrite('file',x,'delimiter',' ')
tic
for k=1:N-1
    x=rand(1,M);
   %In the real application x is the result of some calculations
    dlmwrite('file',x,'delimiter',' ','-append')
end
toc
tic
for k=1:N
    u=dlmread('file',' ',[k-1 0 k-1 M-1]);
    %calculations using vector u
end
toc

You can test this with any data size you like, I suppose my computer is not nearly as fast as yours at the file processing.

The append feature of fprintf is particularly useful.

Hopefully that helps,

Michael

[Michael Soskind]

Sorry, a few notes

1. I did not remove the dlmwrite at the beginning of the code there.
2. Further, the speed up is quite good for a 1000 element case on my computer.

I have ~2.5 seconds for reading and writing with fprintf, and 5.6 and 60 seconds for writing and reading, respectively with dlmread, using a 1000 element array.

[Chris Koen]

Dear Michael - That does indeed help, thank you very much!

I made one change to get each x on a different line:

    fid = fopen('file2.txt', 'a');
    q=strcat( repmat('%f ', 1,M),'\n');
    for k=1:N
        x=rand(1,M);
        fprintf(fid,q, x);
    end
    fclose(fid);