Filter löschen
Filter löschen

How to speed up a parfor loop with large broadcast variables

13 Ansichten (letzte 30 Tage)
I have a calculation to do on a large list of points, that makes use of two large cell arrays (~1.5 GB). My efforts in changing the main loop of the code from for to parfor seems to slow it down by a factor of the number of workers (all on a local machine).
The code I am trying to run looks like:
Main body:
idx_vals = {cell array of size n};
W_vals = {cell array of size n};
idx = parallel.pool.Constant(idx_vals);
W = parallel.pool.Constant(W_vals);
out_val = func(idx, W);
Function call:
function val = func(idx, W);
n = length(idx);
val = zeros(1, n);
parfor k=1:n
idx_k = idx.Value(k);
idx_k = idx_k{1};
W_k = W.Value(k);
W_k = W_k{1};
val(k) = [some function of idx_k and W_k];
Some possibly useful comments:
  • idk and W are cell arrays containing matrices as elements. I am using a cell array because the size of the matrix changes from element to element.
  • I also looked at making idx and W global variables, but the compiler gave warnings about that being a bad idea.
  • I also tried versions of the code where idx and W were global variables instead of parallel.pool.Constants.
In everything I am trying, it seems that the transfer or access of the large variables idk and W dominates the run time, and that run time is roughly the number of workers * the run time for a for loop instead of parfor.
Any ideas of what I am doing wrong or what else to try?

Akzeptierte Antwort

Edric Ellis
Edric Ellis am 5 Jun. 2019
It seems from your code that you should be able to slice your large variables idx and W. This would mean that each element of those arrays is transmitted only to the worker that needs it, rather than to all workers. Something like this:
% Generate dummy data as cell arrays
idx_vals = arrayfun(@rand, 1:10, 'UniformOutput', false);
W = arrayfun(@rand, 1:10, 'UniformOutput', false);
n = length(idx_vals);
val = zeros(1, n);
parfor k = 1:n
idx_k = idx_vals{k};
W_k = W{k};
val(k) = max(abs(eig(idx_k * W_k)));

Weitere Antworten (0)


Mehr zu Parallel for-Loops (parfor) finden Sie in Help Center und File Exchange




Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by