Using multiple GPUs using MDCS HPC/SLURM cluster with 2 GPUs per node

I have a simulation that requires massive number crunching for which I am currently using HPC/SLURM cluster provided by my university. We have a configuration such that we have 2 GPUs per node hence if I have 15 GPU nodes I should be able to utilize 30 GPUs. Currently I have modified the indpendentSubmitFcn to request SLURM to provide 15 nodes with 2 cores per node (my idea is each task would be put on individual core which can than utilize a GPU). The problem is when I open MATLAB on a single GPU node with local profile and request for 2 workers I am able to utilize two GPUs but when I request workers using MDCS/SLURM I can only access one GPU per node. Is there any way I can programatically request for 2 GPUs per node? This way my simulation could be speeded up by a factor closer to 2.

Answer 1

[Kojiro Saito]

It depends on the integration scripts which you're using.

If you're using support packages from File Exchange, you can change submit arguments by changing "additionalSubmitArgs" in indpendentSubmitFcn.m and communicatingSubmitFcn. I think adding "--gres=gpu:2" will work. For example, in communicatingSubmitFcn.m,

additionalSubmitArgs = sprintf('--ntasks=%d --gres=gpu:2', props.NumberOfTasks);

If you're using customized scripts in your university, you might be able to change the number of GPUs per node from MATLAB command. I suggest you ask your cluster administrators whether you have such options.