I need help with parpool and optimizing recources for image reconstruction, running matlab with batch script
2 Ansichten (letzte 30 Tage)
Ältere Kommentare anzeigen
Im running an image reconstruction (1798 head volumes) script on a computation cluster
These are my recources
Nodes: 1
Mem per core: 43 GB
Cores per node: 24
Mem tot: 1024 GB
The reconstruction script reconstructs each full head volume (total 1798 head volumes) in the for loop for k=1:length(tframes), tframes=1798
:
![](https://www.mathworks.com/matlabcentral/answers/uploaded_files/341170/image.png)
Now, my idea is to use parpool to parallelize the for loop. When using parpool one needs to specify the number of workers. This number should match the number of CPUs requested.
IN the computation cluster I use recources are requested as follows:
#SBATCH -p hugemem % name of the partition
#SBATCH -t 072:00:00 &time
#SBATCH --nodes=1 %nodes
#SBATCH --mem-per-cpu= 43G
#SBATCH --ntasks= 24
#SBATCH --cpus-per-task= 1
My question, how should I use parpool to parallelize this task of reconstructing the 1798 head volumes? Or do you think in this case parfor would be better?
0 Kommentare
Antworten (1)
Raymond Norris
am 5 Aug. 2020
Hi,
I haven't taken the time to see if parfor will work in your case, but have some general comments/thoughts.
parfor is a MATLAB parallel construct for parallelizing a for loop. In order for a parfor to run correctly, it need a pool of workers (i.e. parpool). By default, if a parallel pool is not running, calling parfor will instantiate the pool with a default number of workers (12 in your case).
I would suggest the following. Change your Slurm jobscript as such
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=24
You're only running 1 task (MATLAB), but you'd like to dedicate 24 cores to it(your pool). In your MATLAB code, add the following
% Query for available cores
sz = str2num(getenv('SLURM_CPUS_PER_TASK'));
if isempty(sz), sz = maxNumCompThreads; end
p = parpool('local',sz);
parfor k = 1:length(tframes)
...
end
You might decriment sz by 1 to account for the MATLAB client running.
Raymond
0 Kommentare
Siehe auch
Kategorien
Mehr zu Parallel Computing Fundamentals finden Sie in Help Center und File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!