working with large matrices

I have to work with large matrices (e.g. A = 8000 x 100.000, all non-zero values).

I want to calculate T= ((A'*A) + lamda*speye(n))\(A'*A); with lamda =e.g. 1-e-3.

I have installed 64 gb ddr and as expected I run out of memory (also when I introduce a threshold and force A to be sparse, and then create sparse(A) and sparse (A') and try to calculate T.

Are there alternative ways to calculate T without out of memory issues?

[David Goodmanson]

Hello Bram,

If A is 8e3 x 1e5 then T is going to be 1e5 x 1e5, is that correct?

[Bram Stegeman]

Hello David,

That is correct. 1e5 x 1e5 could already results in memory issues is my experience.

Forget to mension that I'm only interested in the diagonal values of T.

[Stephan Koschel]

You are only interested in the diagonals of a matrix multiplication?

I would implement an iteration over the diagonal elements and load the corresponding columns and rows from the two matrices. The entry on the diagonal becomes something like sum(current_row .* current_col)

The iteration could slow down the process, but you only need to load two vectors into memory.

Answer 1

[Munish Raj]

Hello Bram,

You could look at implementing Tall Arrays.

The documentation link can be found here.