work with large matrix by using memmapfile

I would like to map a very large matrix to memory to work with it. For example, I have a matrix of real numbers of size 100000*100000. I can not make this matrix, but I have to work with it. Does anyone know how to make this matrix using memmapfile and doing process on it?

Thank you very much

Answer 1

[Walter Roberson]

To use memmapfile you need to have as much memory as the size of the file. The exception to this is if, at the time of mapping, you specify Offset and you specify a Repeat on your Format that implies that only a limited amount of memory is to be used.

I suspect you might find a matFile() more useful for your purpose, provided that you are extracting sub-sections of the array and working with them and writing new sub-sections. If you need to do something like a matrix multiplication of the entire array then you will need to start using swap space, preferably swap space located on an SSD.

With a matrix that large, we would wonder whether perhaps you should be using sparse matrices.

[sh as]

Thank you very much for quick reply.

In fact, at first I make a sparse matrice because just a few elements are filled in with real numbers. Then, I have to complete the whole elements with some "matrix completion algorithms" (that is a work in linear algebra and NLP). So, I need all elements of matrix! although I do not know yet how to modify my matrix completion algorithm to work with just a part of the matrix (not whole at the same time), I wonder if I could save the matrix in memory using memmapFile. I will also try matFile()...