out memory problem

hi, My array of data is 211,231*4,when process it I got this error:

Out of memory. Type HELP MEMORY for your options. keep in mind, I use clear ,close matlab and open it what I have to do to get over this error.

Thanks in advance

Answer 1

[Jan]

Do you mean a [211'231 x 4] array? This array has 6.7 MB only, such that the must be another large array, which occupies much more memory.

Please post the corresponding source code, which is used to the create the array.

[huda nawaf]

thanks:

I read from file , then print it to array

the file with size 4.61 MB. the problem that the running time one hour. I don't know where is the problem

this is code:

f=fopen('ws.data','r+');%%%%%% open the original data file

b1=fscanf(f,'%d');

b=sprintf('%d\n', b1);

fclose all

L=length(b);

k=1;i=1

while k<=L

for j=1:4

mat(i,j)=b(k);

k=k+1;

end

i=i+1;

end

for k2=1:i-1

k1=1;

for j=1:2:4

mat1(k2,k1)=mat(k2,j);

k1=k+1;

end

end

[Jan]

What do you want to achieve? Please search in this forum and the net for the term "pre-allocation".

[Walter Roberson]

Please take some time, Huda, to read about "vectorization" in MATLAB. You have been using MATLAB for some time now; it is time for you to study that topic and put it in to practice.

And, like Jan says, pre-allocate.

Answer 2

[Walter Roberson]

Is that 211231 x 4, each entry a double precision number (8 bytes), or is it 211 x 231 with each entry a 4 byte number, or is it 211 x 231 x 4, each entry a double precision number (8 bytes)?

If it is 211231 x 4 double-precision entries, that should be only 6.4 megabytes; if you are unable to allocate a 6.4 megabyte array then you are too short on memory to do anything practical.

Please show the exact call you use to allocate the memory.

For example, if you were using

A = zeros(211231*4)

then we would immediately diagnose the problem as being that when you supply only a single scalar size to zeros() (or ones(), or rand()) then a square matrix is constructed that is that number of rows and columns.

[huda nawaf]

thanks:

I read from file , then print it to array(no. of rows are 211231, and columns are 4)

the file with size 4.61 MB. the problem that the running time one hour. I don't know where is the problem

[huda nawaf]

Thanks,I will check pre-allocate. As you asked me, I ran A(1:211231,1:4)=1

,there is no problem,it is not take time to run.

please help me , below the code again:

f=fopen('ws.txt','r+');%%%%%% open the original data file

b=fscanf(f,'%d');

fclose (f)

L=length(b);

k=1;i=1

while k <=L

for j=1:4

mat(i,j)=b(k);

k=k+1;

end

i=i+1;

end

[Walter Roberson]

This is pretty much the same problem I solved for you over in http://www.mathworks.com/matlabcentral/answers/17949-problem-in-sprintf

the whole code after the fclose() can be replaced with a reshape().

[huda nawaf]

I change my computer, now my computer has RAM with 4 Gb instead of the old that has 1.8 Gb , unfortunately , the same problem

with this code:

the structure of files as

ex.

(1,1) 1000

(2,1) 12

.

.

.

fid = fopen('d:\matlab\R2011a\bin\flixsterdata\finalflix0.txt','r');

c = textscan(fid, '(%d,%d)\t%d');

c1=double(c{1});c2=double(c{2});c3=double(c{3});

L=length(c1)%%%L= 2444547

w=double(c1(1:L)); w1=double(c2(1:L)); w2=double(c3(1:L));

flix0=accumarray([w,w1],w2);

please, help me what I have to do?

[Walter Roberson]

It might help to stop making so many copies of variables.

fid = fopen('d:\matlab\R2011a\bin\flixsterdata\finalflix0.txt','r');

c = textscan(fid, '(%g,%g)\t%g');

fclose(fid);

flix0 = accumarray([c{1}, c{2}], c{3}]);

By the way, what is max(c{1}) and max(c{2}) ?

[huda nawaf]

thanks,

max(c1)=50000

max(c2)=30977

length(c1)=2444547

[Walter Roberson]

Urrr, you will definitely want to use a cell array or sparse matrix for that -- accumarray would need about 12 Gb to store that matrix in non-sparse form.

[huda nawaf]

I did what you suggested regarding variables

the problem is still.

what about dynamic array?

can solve my problem?

[Walter Roberson]

I never looked up the storage requirements for sparse arrays.

What do you mean by "dynamic array" in this context?

[huda nawaf]

someone who is working with aother language suggest to work with dyn. array, I'm not familiar with it,ut i will read about it if it is in matlab

walter, please tell me how I can use cell array to solve my prolem

give me example please.

thanks

[Walter Roberson]

I never did find out what problem it is you are really trying to deal with -- though I did ask a few times.