'parfor' Memory Usage Greater Than Expected

Just before the execution of a particular loop in my code the memory usage will be ~6GB.

If I use 'for' then the memory usage will increase by ~600MB during execution of a single loop and drop back down.

If I use 'parfor' with just 2 workers the memory usage will exceed my 32GB of RAM and 32GB of swap.

Here is the loop:

parfor n=1:length( loop_Unique_Handle_Object_Array )
   result{n} = someFunction( shared_Data,...
                             loop_Unique_Handle_Object_Array(n) );
end

'shared_Data' is 100MB

Each instance of loop_Unique_Handle_Object_Array is 200MB

I dont understand why the amount of RAM required by 'parfor' is greater than the amount required by 'for' * numWorkers + Overhead. Is this a result of using handle objects?

Answer 1

[Matt J]

Is this a result of using handle objects?

Possibly. All data that a handle object points to will be cloned and transmitted to the workers. It's not clear how you would have measured the memory footprint of everything your handle object points to and concluded that it is 200MB. MATLAB doesn't have any direct tools that would let you measure this footprint (whos() will not be reliable, in particular).

An easy test would be to just do a simplified parfor which only broadcasts the handle objects

    parfor n=1:length( loop_Unique_Handle_Object_Array )
       result{n} = loop_Unique_Handle_Object_Array(n);
    end

[Matt J]

You should also be careful if any of your objects contain anonymous function handles. They can contain handle references to the entire workspace, which will get cloned during transmission to parallel workers, see

http://www.mathworks.com/matlabcentral/answers/115254-parfor-behavior-sensitive-to-comments

[Jay]

I am basing the size of the handles on their size when they are cast to a struct. If I remember correctly there is at least one function handle in the object that references the work space.

In either case, for two workers, why would parfor require more than double the amount of ram being used by the entire simulation? Does it start getting the memory ready for loop iterations that have yet to occur?

I will perform the requested tests when I am back at the machine on Monday.

[Matt J]

In either case, for two workers, why would parfor require more than double the amount of ram being used by the entire simulation?

I don't know how you're measuring RAM consumption (Task Manager, WHOS, MEMORY?) and therefore don't know if the memory footprint numbers you've given so far can be trusted.

Beyond that, you presumably have more than 2 handle objects in your array loop_Unique_Handle_Object_Array that you are transmitting to the workers. I vaguely wonder if each individual one of those is having its memory cloned. That would be terribly unnecessary and inefficient on the part of parfor, but might be something worth testing nevertheless. It might also be version dependent. Is this R2014?

[Jay]

Memory usage is being measured with htop

loop_Unique_Handle_Object_Array is between 10 and 100 elements long.

MATLAB is 2013b.

If I make loop_Unique_Handle_Object_Array be 10 elements long the memory usage as measured by htop just before entering the following loop will be 1.4 GB. When I open a parpool of size 2 memory usage will increase to 1.7GB. During the loop memory usage will reach 6.4GB, after the loop memory usage will be 3.6GB.

If I perform the same test with a parpool of 10 then preloop memory usage will be 1.4GB, after opening the pool its 3.1GB, during the loop it is 5.5GB, after the loop its 5GB.

Note that I am probably missing the actual spike of the memory usage becuase of the limited sampling rate of the htop display.

    parfor n=1:length( loop_Unique_Handle_Object_Array )
       result{n} = loop_Unique_Handle_Object_Array(n);
    end

I have rewritten my code so that it no longer has handle objects in order to avoid any of the other problems mentioned so far, it works far better this way (max ram usage ~5 additional GB).

[Matt J]

Hmmm. Well, if it's due to data cloning, I don't fully understand why the drop from 6.4GB to 3.6 GB.

Another test is to save a small example of loop_Unique_Handle_Object_Array to a .mat file, then reload it under a different name

   save test loop_Unique_Handle_Object_Array
   reload = load('test.mat')

The cloning processes involved in writing to a .mat file are similar to the way data is broadcast in parfor. So, if the file consumes the same amount of memory as parfor and the workspace memory consumption changes in the same way as well, there's reason to think it is due to cloning.

[Jay]

For whatever reason I get an error closing the file when I try to save the handle object array.

I have pretty much convinced myself that its not a great idea to use handle object in parfor, add to that some anonymous functions with references to the workspace and it appears to get real messy.

Besides no longer passing the handle object into the loop, but instead moving the actual required data into sliced temporary variables, I have also started using WorkerObjWrapper.

[Matt J]

That sounds like a good idea. You could also try trimming the memory footprint of the anonymous functions using this FEX file,

http://www.mathworks.com/matlabcentral/fileexchange/45749-clean-anonymous-functions-of-excess-memory

and see if that's of any help.

[Jay]

Excellent, that looks real handy. I just ran into a different problem involving parfor and function handles that I think this may fix.