run background function on subset of allocated parpool nodes on cluster?

2 Ansichten (letzte 30 Tage)
Is it possible to start a parpool, then run a background function on a subset of the parpool's allocated nodes (on a cluster)?
Suppose I allocate (total Procs) = (48 procs) x ( 6 nodes) using
parpool(48*6);
Then can I then invoke some command such as
myNum=[1e6,2e6]
Result1 = parfeval(backgroundPool_1,@rand,1,myNum(1)); % suppose bgPool_1 is a subset of parpool somehow
Result2 = parfeval(backgroundPool_2,@rand,1,myNum(2)); % suppose bgPool_1 is a subset of parpool somehow
where backgroundPool_1 would be like 48*6/2, half of the number of allocated nodes, and same for bgP_2.
  • I would not want to allocate two different parpools, because I am running interactively on a cluster and want all nodes to be pre-allocated on the slurm queue
  1 Kommentar
Edric Ellis
Edric Ellis am 25 Jul. 2022
This isn't possible today. Can you explain more about what you want to do on the pool subset if you could create one? What do you want to run there, and how would it use multiple workers of the subset? One thing I can imagine is doing something like this:
p = parpool(..);
[subPool1, subPool2] = somehowDividePool(p);
fut = parfeval(subPool1, @myFunctionThatCallsParfor, numOuts, in1, in2);
Here, myFunctionThatCallsParfor has a parfor loop inside, and it gets to use the workers from subPool1 to run that.
Is that the sort of thing you're interested in? If not, it would be useful if you could explain what you would like to be able to do.

Melden Sie sich an, um zu kommentieren.

Antworten (0)

Kategorien

Mehr zu Parallel Computing Fundamentals finden Sie in Help Center und File Exchange

Produkte


Version

R2019a

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by