Removing linearly dependent rows (or linearly dependent columns) from large matrices (bigger than 250,000 x 250,000)

I need to remove linearly dependent rows from large sparse matrices (> 250,000 rows and columns). The recommended solutions online call for using "chol", "qr", or "ldl" for this purpose (rref is also supposed to work, but some have advised against it).

• I have tried these approaches on a linux machine running R2015a with 64GB RAM. Unfortunately, I get an "out of memory" error when calling chol, qr, or ldl. rref does not produce such an error, but it is prohibitively slow for the problems I am considering.
• I have also tried these approaches on a mac running R2015b with 8GB RAM. No functions give out-of-memory errors, but the mac OS does allocate 10GB of virtual memory (even though the mac has an SSD, this is still prohibitively slow).

I am looking for a solution satisfying (1) faster performance than rref, and (2) it does not generate out-of-memory errors on the linux machine running R2015a.

Does anyone know of such solutions?

Other question (not as important as question above):

It is very odd that a machine with 64GB RAM generates an out-of-memory error while a machine with 8GB RAM does not. The differences between the machines are that the machine with 64GB RAM is running R2015a on linux, while the 8GB machine is running R2015b on OSX 10.10 (Yosemite). Are there known memory management issues with R2015a, or Matlab linux distributions?

Answer 1

[Arnab Sen]

Hi Riley ,

From MATLAB preference you can try increasing the JAVA Heap size and see if the issue gets resolved.

For this do the following:

1.Go to Preference from MATLAB.

2. In the preference window select 'General ->Java Heap Memory'.

3. Use the slider to allow access larger JAVA Heap memory which would be dynamically allocated at runtime.

Following links might help to figure out why Linux machine getting 'out of memory error' despite large RAM size

http://www.linuxdevcenter.com/pub/a/linux/2006/11/30/linux-out-of-memory.html?page=2

MATLAB has some methods to figure out the real issue. Refer to the following link:

http://www.mathworks.com/help/matlab/matlab_prog/resolving-out-of-memory-errors.html

[Riley Murray]

Thanks for your response Arnab! I will look into Java heap size as you suggest.

Although -- it is not clear to me for why Java heap size would affect memory issues when calling numerically intensive codes such as "chol" and "qr" (which are sure to be implemented in C, not Java). Do you have any reasoning behind why the Java heap size would be relevant in this case?