- Suppress the output arguments you don't need
- Save to a buffer and save that to disk when it becomes too large.
- Have each worker save its own output, not aggregating. You can put all the files together yourself afterwards
- Use sparse arrays
Info
Diese Frage ist geschlossen. Öffnen Sie sie erneut, um sie zu bearbeiten oder zu beantworten.
parfor loops with large outputs
2 Ansichten (letzte 30 Tage)
Ältere Kommentare anzeigen
Hi,
I'm using MATLAB's parallel toolbox on a shiny server (32 physical cores, 64 logical, 512GB RAM). All my parallelization needs can be classified as 'embarrassingly parallel', e.g.:
parfor i_par = 1:n_tot
[CalcOutputs_array(:,:,i_par), AdvancedCalcOutputs_array(i_par)] = CPU_DemandingFunction(InputsStruct_array(i_par));
end
In order to minimize overheads, i've already split the input such that each worker receives only what it needs.
The output, however, can become quite large (1GB-10GB), and data collection from the workers is vastly over-weighting the calculation itself.
I'm positive that this is the case since by lowering the amount of data outputted from the function (i have a flag that controls it) - the runtime improvement factor approaches the # of cores i'm using.
Can that be improved? Is there any remedy for data-collection overheads?
Thanks a lot!
Yanir
1 Kommentar
José-Luis
am 30 Jun. 2016
This is a very general question.
You could:
etc...
Antworten (0)
Diese Frage ist geschlossen.
Siehe auch
Produkte
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!