Out of memory issue

1 Ansicht (letzte 30 Tage)
Amit Chakraborty
Amit Chakraborty am 20 Sep. 2021
Kommentiert: Amit Chakraborty am 21 Sep. 2021
I have a sparse matrix which has a size of, (2031616x4096 sparse double).
I want to make 10 subset of this matrix and need to convert this matrix in to "full". So, I have made 10 subset and while processing it was able to covert 5 subset into " full" and for the rest of the five subset it is showing me the message : "out of memory".
One thing I have done that is, when I saved the each subset separately into a external harddisk may be at that time it will be possible but here again the problem appears during the importing the subset matrix. I know that it is not a solution .
I will appreciate if anyone can suggest some helpful solution to tackle this problem.
  3 Kommentare
Walter Roberson
Walter Roberson am 20 Sep. 2021
It sounds as if you will have multiple subsets in memory ?
Amit Chakraborty
Amit Chakraborty am 20 Sep. 2021
The size of each subset is : 196608x4096 double

Melden Sie sich an, um zu kommentieren.

Antworten (1)

Steven Lord
Steven Lord am 20 Sep. 2021
sz = [196608 4096];
numberOfElements = prod(sz);
bytes = 8*numberOfElements;
gb = bytes/1024^3
gb = 6
Each subset is 6 GB in size (assuming real doubles, 12 if complex doubles) when converted into a full array. Depending on how much memory you have available it's quite possible you couldn't fit all ten subsets in memory at once.
Can you avoid the conversion to a full matrix? MATLAB can save sparse matrices to MAT-files and load them from those files perfectly well. Alternately if you need to store them in a format some other program can read perhaps write a coordinate list (which you can generate using the find function) to the disk?
Alternately clear-ing each subset once it has been written to disk may allow MATLAB to reuse that memory for the next subset.
  1 Kommentar
Amit Chakraborty
Amit Chakraborty am 21 Sep. 2021
** All the subsets are with real value no complex value is there.
** I need to convert the sparse matrix into "full" because i need this for further calculation.
** If it is possible for you to show me an example of writing coordinate list with "find" function?
** Lastly, yes I have been clearning the unwanted variable once I worked with it so that MATLAB can reuse the memory.
Thanks to Everyone !

Melden Sie sich an, um zu kommentieren.

Kategorien

Mehr zu Sparse Matrices finden Sie in Help Center und File Exchange

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by