Multi-level parallelization

I am running on a machine with 64 cores. My code can run in parallel efficiently only on about 12 workers. When I give it more than 12 workers, the code actually slows down.

However, I can sumultaneously run two instances of the same code (for parametric studues) by starting up two sessions of MATLAB and running one code on each session. Because the two codes do not interact and do not communicate with each other, they each seem to run independently at full speed without sacrificing parallel efficiency.

QUESTION: Can I do this from one session of MATLAB? Ideally I would want some kind of multi-level parallelization, which would look something like this proto-code:

pool1 = parpool("local1", 12);

pool2 = parpool("local2", 12);

result1 = runCodeInBackground(pool1, "myCode", parameterForParametricStudy(1));

result2 = runCodeInBackground(pool2, "myCode", parameterForParametricStudy(2));

I realize that the "batch" command is designed for this. However, batch involves a lot of overhead in starting up each batch job. I would like to set up two (or more) independent parallel pools at the beginning, and then use these existing pools interactively.

Answer 1

[Raymond Norris]

MATLAB does not support multiple parallel pools in the same session. With that said, as you've pointed out, batch will do what you want and has almost the same signature as runCodeInBackground.

result1 = runCodeInBackground(pool1, "myCode", parameterForParametricStudy(1));

versus

job1 = batch("myCode", nout, {parameterForParametricStudy(1)}, 'Pool',12);

Where nout is the number of output arguments of myCode you want returned.