Specify the parallel pool job timeout
25 Ansichten (letzte 30 Tage)
Ältere Kommentare anzeigen
Hi,
I am running some tests using a remote cluster (no local cluster). I send my functions in batch mode. I know that the functions take a large execution time, around 2 / 4 hours. When I try to run, I get the message:
'The parallel pool job was cancelled because it failed to finish within the specified parallel pool job timout of 300 seconds'
I looked into the documentation how to change the default timeout time. The only way I could find is with the "wait" command, as
wait(job,"finished",18000);
However, I keep on getting the same error. How can I change the default parallel pool job timeout in the remote cluster?
0 Kommentare
Antworten (1)
Raymond Norris
am 1 Okt. 2021
So you doing something like the following
cluster = parcluster;
job = cluster.batch(@mycode,...., 'Pool',size);
Then what you're suggesting is that let's say your code looked something like
function mycode
pause(10 * 60)
parfor idx = 1:N
...
end
On the cluster, the workers have a default timeout of 5 minutes and therefore errors out because you're running code (the pause) for 10 minutes before the workers are being used in the parfor.
'The parallel pool job was cancelled because it failed to finish within the specified parallel pool job timout of 300 seconds'
I tried quickly reproducing this with the local scheduler, but didn't (it shouldn't matter if I'm jusing local). How are you getting the error message? And which scheduler are you using MJS or generic (e.g. PBS)?
9 Kommentare
Raymond Norris
am 2 Okt. 2021
One of the workers crashed, possibly because of an out of memory issue. Email Technical Support (support@mathworks.com) and they can walk you through debug steps and getting the MJS log files to troubleshoot.
Siehe auch
Kategorien
Mehr zu MATLAB Parallel Server finden Sie in Help Center und File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!