Memory management and allocation

Hi everyone,

I'm running 64-bit Matlab R2012b on a 64 bit linux machine. The machine has 16 cores and a total of 64GB RAM. I'm running this particle simulation for which I'm using arrays that are 2X67108864 in size. I checked with the whos function and my total memory usage at any given time is about 8 GB. I'm trying to parallelize this code using spmd, but when it tries to create three instances of this code (which should at most be 24GB), it gives me the following error:

Error using distcompserialize
Error during serialization

Error in spmdlang.RemoteSpmdExecutor/initiateComputation (line 82)
                fcns  = distcompMakeByteBufferHandle( ...

Error in spmdlang.spmd_feval_impl (line 14)
    blockExecutor.initiateComputation();

Error in spmd_feval (line 8)
        spmdlang.spmd_feval_impl( varargin{:} );

Through my online research I'm led to believe that this is a memory problem. My questions are the following:

1. Since "memory" is not an available option on Linux, is there any other way to find out memory allocation to Matlab. 2. Is there a way to increase allocation to Matlab because it seems to me that it's not using all that is available (which it should since it's 64 bit matlab on 64 bit linux). 3. Is this a different problem (not memory) altogether?

Thanks,

Siddharth

Answer 1

[Walter Roberson]

On MS Windows, serialization is restricted to 4 gigabytes (I think it is) even on 64 bit MATLAB versions. I do not know why: possibly it uses embedded 32 bit addresses. The same issue might be at play here.

Are you moving some large objects around? Especially if you are moving the same large object to several workers, there is a container class available that can act to delete the copy until runtime.

[SiddharthKrishnamoorthy]

Yes well I'm executing this function on two workers that access a variable that's about 1 GB in size and produce a variable each that's 1 GB in size.

[Walter Roberson]

Have a look at http://www.mathworks.com/matlabcentral/fileexchange/31972-worker-object-wrapper

[SiddharthKrishnamoorthy]

Will do and get back, thanks!

[SiddharthKrishnamoorthy]

Hi, I checked this wrapper out. In my code I still need to gather data from all workers back to the client and then execute some operations on the data. Do you think it would still help? It seems like this wrapper works when you can leave the data on the workers...

[Walter Roberson]

Your difficulty reported earlier is a serialization error, which could be either in sending data to the worker or in sending data back. I think, from the message, it is in sending data to the worker. So if you can cut the amount of data you send to the worker, you might be able to avoid the error.

I am guessing here that you are needing more data sent to the worker than you need to get back. If, though, you are needing to send in an initial (nontrivial) value of a big matrix and have it modified and sent the modified version back, then I do not know if the wrapper will be of benefit.

Note: I am working on general parallel principles here, together with an item or two that I have picked up reading some responses. I have never had access to the toolbox myself.

[SiddharthKrishnamoorthy]

Yes well I'm working on a code that interacts between a fixed grid and moving particles. The codes I want to parallelize are interpolations between particle locations and grid nodes. While the matrix containing grid nodes is small, the matrix containing particle locations is very large. So in one case I send a huge non trivial matrix and get back a relatively small one after interpolation and in the other I send a large matrix and get back a large one. I've also found that with fewer particles when I don't get the serialization error, the time taken in the parallel implementation is more than the serial one - I'm guessing this is because of the large amount of data transfer between client and workers, therefore it would be highly desirable to get rid of it as much as possible.

[Walter Roberson]

For the interpolation of particles to grid, I wonder if you are using a 2D histogram technique such as http://www.mathworks.com/matlabcentral/fileexchange/9896-2d-histogram-calculation ? This would depend upon the grid being rectangular but not on them being equally spaced in x or in y. If they are evenly spaced in x and y I think there might be a faster approach -- but I would need to benchmark whether it was indeed faster. Ah, I see that you could also use http://www.mathworks.com/help/matlab/ref/griddedinterpolantclass.html

When you are doing grid to particles, I presume that the particle matrix is not sorted. I would have to think further about appropriate algorithms.

Have you considered using quadtrees ? Whether that would be cost-efficient would depend on how much you are moving the particles. Building the quadtree from the particle locations would possibly be a bit expensive if all you are going to do with it is find a small number of grid points within the matrix; it would be best suited for the case where the quadtree, once built, is queried repeatedly. Also, if particles do not move far compared to the typical distances between them, then updating the quadtree can be comparatively efficient.

Anyhow, there is a possibility I might be able to think of something useful if you could say more about the properties of the particle matrix and the grid and how you use them.

[SiddharthKrishnamoorthy]

Well so the interpolation routines themselves have been written in C++, and I'm using mex to interface between matlab and C++, that's actually given me great amount of time saving (~12-20 times faster than before). The particle matrix is 2XNp (Np being the number of particles (large), x and y locations). There is also a velocity matrix. Now the interpolation needs to be done for three species, but can be done independently. What I was trying to do was to pass the grid locations and the species particle locations and getting the interpolated charge values on the grid. I was trying to use 3 processors simultaneously to carry out this process. The problem is, passing these position and velocity matrices around is taking a bunch of time, even if the interpolation itself happens faster.