Can a parfor loop be restarted?
2 Ansichten (letzte 30 Tage)
Ältere Kommentare anzeigen
I am processing large amounts of files. Each file might take thirty minutes to an hour. Here's the code snippet on the high level:
parfor firstfile:lastfile
[morestuff1, morestuff2] = runfunction(stuff1, stuff2);
end
Problem is that every night I leave it running on the server, something wrong happens and the parfor doesn't complete. I want to be able to catch a loop that fails for any reason and restart it later. Here's what I am proposing as the solution:
files = firstfile:lastfile;
while length(files) > 0
parfor currentfile = files
try
[morestuff1, morestuff2] = runfunction(stuff1, stuff2);
files(files == currentfile) = [];
catch
% not sure what to do here
end
end
end
0 Kommentare
Antworten (1)
OCDER
am 11 Jun. 2018
Perhaps you can save the input variables used for your runfunction into a separate cell array. Try this for example:
ErrorFile = cell(1, 1000);
parfor k = 1:1000
stuff1 = rand(1);
stuff2 = rand(1);
try
assert(stuff1 > stuff2, 'OOPS: Error in this function!'); %Pretend this is the "runfunction". If there's an error, record the input variables.
catch Msg
ErrorFile{k} = {Msg, k, stuff1, stuff2}; %Save the iteration #, variables, and error message
end
end
ErrorFile(cellfun(@isempty, ErrorFile)) = []; %This stores variables required to redo parfor with errors
cellfun(@(x) display(x{1}), ErrorFile); %Display all the error messages
0 Kommentare
Siehe auch
Kategorien
Mehr zu Startup and Shutdown finden Sie in Help Center und File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!