Handling memory issues by connecting to database and retrieving only required data

I have an excel file as large as 20MB with roughly abt 1.5 million rows. I run out of memory while I perform any sort of operation on the data though I am able to read data completely from this huge file. As an alternative would it be a good idea to connect to a database and fetch only the data required for each of my operation (Although this could mean connecting to the database several number of times.) ?

Answer 1

[Walter Roberson]

It seems odd that you would run out of memory on basic operations on such a small file. You are only averaging about 12 characters per row, which would tend to suggest one double precision number per row, which would add up to only 12 megabytes of storage. That's relatively small.

Please check that you are not accidentally using "*" (matrix multiplication) instead of ".*" (element-by-element multiplication) in your routines.

Does even something as simple as

B = A + 0;

cause you to run out of memory? If not then just read the file once and access the sub-ranges you need. If it does run out of memory with something that simple, please use "whos" to figure out what is taking up so much memory. If you are working on the 32 bit Windows version, the command "memstats" can tell you about available memory.

Answer 2

[Ms. Mat]

Yes. Your right. The file has just 3 columns with 2 numeric and 1 string column. I increased the virtual memory and the problem got solved. I observed that when I ran memstats even after clearing the space a lot of memory was held. It takes a while before the memory is freed up.

Thank You