How to make data persist on workers between calls to parfor?
1 Ansicht (letzte 30 Tage)
Ältere Kommentare anzeigen
Essentially I want to evaluate a loop of the following form as quick as possible:
for k = 1:K
S = 0;
for n = 1:N
S = S + f(a,X{n});
end
a = g(a,S);
end
where 'f' and 'g' are functions, 'X' is an N-by-1 cell array (~1GB) with N large and 'a' is a small numeric array (<1kB).
Using parfor in the inner loop will speed it up but surely it would be even faster if the data 'X' persisted on the workers between calls to parfor. It's just I have so far failed to get codistributed arrays and spmd to work as I'm sure they're intended. If someone could provide an explicit example for this special case I'd be very grateful.
0 Kommentare
Antworten (1)
Edric Ellis
am 17 Mär. 2014
This looks like it might be a good case for the Worker Object Wrapper. The examples there should show you how to use it - but basically you need to wrap 'X' and then extract the Value field inside the PARFOR loop.
4 Kommentare
Edric Ellis
am 18 Mär. 2014
Here's how to use WorkerObjWrapper in your first example. (For reasons that aren't entirely clear to me, this actually slows things down - perhaps your actual code will exhibit speedup though if the amount of memory to be transferred becomes more significant)
L = 8; N = 400; testdata = WorkerObjWrapper(rand(L,L,N));
tic
result_par = zeros(1,N);
for k = 1:10
parfor n = 1:N
v = testdata.Value;
result_par(n) = result_par(n) + norm(v(:,:,n));
end
end
toc
I think your second example is a bit confused about whether you should be dealing with distributed or codistributed arrays. You've made 'testdata_dist' be 'distributed', but 'redist' is 'codistributed'. In general, you should deal with only 'distributed' outside SPMD, and 'codistributed' inside. The data types are automatically transformed when you cross the SPMD boundary. Also, for-drange should only be used inside an SPMD context. For reference, here's what I think you should be doing there - but indexing distributed arrays is rather slow, so this does not achieve good performance:
tic
testdata_dist = distributed(testdata);
resdist = distributed.zeros(1, N);
for k = 1:10
spmd
% inside this block, 'testdata_dist' and 'resdist'
% are both transformed to 'codistributed'.
for n = drange(1:N)
resdist(n) = resdist(n) + norm(testdata_dist(:,:,n));
end
end
end
result_spmd = gather(resdist);
toc
Siehe auch
Kategorien
Mehr zu Distributed Arrays finden Sie in Help Center und File Exchange
Produkte
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!