Find optmal number of parallel workers (depending on machine memory)
5 Ansichten (letzte 30 Tage)
Ältere Kommentare anzeigen
Matlabber 1.0
am 26 Jul. 2015
Kommentiert: Walter Roberson
am 27 Jul. 2015
Dear all, I am running some Image analysis in a parfor loop. is there any way to know before running, what the optimal number of workers is (run time vs. out of memory error)? or even better, adjust the number of workers dynamically?
For my task, memory is the bottleneck not the CPUs. If i have too many workers, I run into the out of memory error.
Based on what I see in the Task manager / performance tab, for 3 workers my memory oscillates heavily between 40 and 90% at the beginning of the job. minutes later, I see the used memory getting more steady at about 50-60% memory. SO I wonder if I could not add another worker or two at that stage
Thanks for any advice!
1 Kommentar
Walter Roberson
am 27 Jul. 2015
It can be shown in theory that this cannot always be handled automatically.
If your memory reduces after peaking then the implication would seem to be that some of the early workers reads a large file or constructs a large array not used by the later workers -- it is difficult for programs such as MATLAB to "give back" memory to the operating system so it typically only happens when a worker exits (interactively there is the pack() command). You might perhaps be able to split the workers up, doing the large workers in one group, then increasing the pool size to do the smaller workers.
Akzeptierte Antwort
Edric Ellis
am 27 Jul. 2015
This is a tricky problem to handle automatically - I think your best bet is simply to time and observe the behaviour on your system and then try to choose the number of workers that gives the best performance.
0 Kommentare
Weitere Antworten (1)
Matlabber 1.0
am 27 Jul. 2015
1 Kommentar
Walter Roberson
am 27 Jul. 2015
Are the image files different sizes, or is different processing done? If everything is the same size then there should be no peaks except for going from initial conditions to reading in the file.
Siehe auch
Kategorien
Mehr zu MATLAB Parallel Server finden Sie in Help Center und File Exchange
Produkte
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!