File Exchange

image thumbnail

PARFOR progress monitor (progress bar) v4

version 2.0.4 (53 KB) by Frerk Saxen
Progress monitor for matlab parfor (parallel) loops with estimation of the remaining time

49 Downloads

Updated 05 Sep 2019

From GitHub

View Version History

View license on GitHub

A very ressource efficient Matlab class for progress monitoring during a `parfor` loop displaying the remaining time and optional progress of each worker.
It supports distributed worker pools (i.e. doesn't only work on local pools).

Usage:
% 'numIterations' is an integer with the total number of iterations in the loop.
% Feel free to increase this even higher and see other progress monitors fail.
numIterations = 100000;

% Then construct a ParforProgressbar object:
ppm = ParforProgressbar(numIterations);

parfor i = 1:numIterations
% do some parallel computation
pause(100/numIterations);
% increment counter to track progress
ppm.increment();
end

% Delete the progress handle when the parfor loop is done (otherwise the timer that keeps updating the progress might not stop).
delete(ppm);

Optional parameters:
ppm = ParforProgressbar(numIterations) constructs a ParforProgressbar object.
'numIterations' is an integer with the total number of
iterations in the parfor loop.

ppm = ParforProgressbar(___, 'showWorkerProgress', true) will display
the progress of all workers (default: false).

ppm = ParforProgressbar(___, 'progressBarUpdatePeriod', 1.5) will
update the progressbar every 1.5 second (default: 1.0 seconds).

ppm = ParforProgressbar(___, 'title', 'my fancy title') will
show 'my fancy title' on the progressbar.

ppm = ParforProgressbar(___, 'parpool', 'local') will
start the parallel pool (parpool) using the 'local' profile.

ppm = ParforProgressbar(___, 'parpool', {profilename, poolsize, Name, Value})
will start the parallel pool (parpool) using the profilename profile with
poolsize workers and any Name Value pair supported by function parpool.

Benefits:
1. It's the first parfor progress monitor that also displays the remaining time.
2. It's the first parfor progress monitor that also displays each workers progress.
3. It scales from very small number of iterations to arbitrarily high number of iterations with a very small footprint.

Drawbacks:
1. It does slow down the computation. How much? It depends on how often you update the progressbar (on default every 1.0 seconds - but this is a parameter you can adjust).
Updating the progressbar on my computer takes 40ms on average. i.e. one of the x workers updates the progressbar (by default every second) and spends an additional 40ms every second = 4%.
But you have x-1 workers that don't get delayed at all (calling increment has a neglegible effect even for millions of iterations).
2. It requires the Instrument Control Toolbox

Difference to 60135-parfor-progress-monitor-progress-bar-v3:
1. Using [progressbar](https://de.mathworks.com/matlabcentral/fileexchange/6922-progressbar) with it's nice drawing of the remaining time.
2. Complete matlab implementation, no Java.
3. Each increment, Dylan's java based implementation connects via tcp to the server and closes the connection immediately without sending any data.
The server increments the counter just based on an established connection.
This is quite fast but for very short loop cycles (like the above) it results in way too many connections.
The original ParforProgMonv3 solves this by letting the user choose a stepSize manually. However, this is combersome and non-intuitive.
This update calculates the stepsize automatically and thus maintains a very fast execution time even for very short loop cycles.
4. Instead of tcp socket we use a udp socket which is established on construction and not opened/closed at each loop cycle.
5. To track each worker progress, each worker sends its own progress to the server via udp.
6. Small interface changes: I don't really care about the window title of the progress bar. This is now an optional parameter and now also properly monitored by matlab's input parser.

Cite As

Frerk Saxen (2021). PARFOR progress monitor (progress bar) v4 (https://github.com/fsaxen/ParforProgMon), GitHub. Retrieved .

Comments and Ratings (24)

Afonso Ribeiro

Hey! Great tool, it's been very useful to me. I've been experimenting with thread-based pools recently and I haven't been able to get the progressbar working with it so far, is it not compatible or am I missing something? Thanks

Afonso Ribeiro

Z Wang

Ali Komai

Much faster than other alternatives here on file-exchange. Thank you!

Frerk Saxen

@all: Please file a bug report on github if you have any problems. I'm not often here on file exchange tbh and I do get notifications as soon as a bug report has been filed.
@Christian Steinmetz: I think you would have to simply skip all ppm related calls and call progressbar directly. Theoretically we could add an additional parameter to disable everything, but it would bloat the code without providing huge benefits.
@Xavier Laforgue: It looks like the upd connection has changed since 2016b. I only have 2018b unfortunately.

Xavier Laforgue

Hello,
I can not make the function work: the progress bar(s) appears at the beginning of the loop and disappear when it is done, but they never display any information.
There is a warning that should be the cause:
"The DatagramReceivedFcn is being disabled. To enable the callback property either connect to the hardware with FOPEN or set the DatagramReceivedFcn property.
> In distcomp.remoteparfor/getCompleteIntervals (line 127)
In parallel_function>distributed_execution (line 793)
In parallel_function (line 587)
In testParforProgressbar (line 21) "
However, I have no idea how to fix it.
Could somebody help me, please?
I am using Matlab 2016b and have the Instrument Control Toolbox.

P.S. Initially I was getting the error: "Error using table (line 229)", but I fixed it following the recommendation of using cell arrays instead of character vectors to create the table (I placed the character vectors, arguments of the table function that were not already in cell arrays, in curly brackets)

Jeff Miller

Christian Steinmetz

Great software!

However, I am writing a tool that is sometimes supposed to run in parallel and sometimes not (depending on how much input you give). So I do not always want to start the parallel pool the progress bar is used. However, I can't get the function to run without a parallel pool. For some reason, even if I get the program to go into the increment-function, the receiver-function seems not to be called if I remove the respective lines if I send any fwrite(u.connection, [1, 50], 'ulong')-statement (The 1 would be the worker that I manually have to set to 0 in this case, the 50 would be the value to which the progress monitor should be set). I don't know why this doesn't work.

Frerk Saxen

@max_fire: There is no way to know in advance the exact number of iterations for each worker. It is simply assumed that all do the same work. But some workers do more, others less. While each individual progress is a guess, the total progress however is exact.

max fire

Nice.
But, while some of workers can't achieve 100%, other exceeded that, weird.

Frerk Saxen

@Adam Wyatt: Thank you for your response. If you are willing to contribute you are very welcome to make your changes and do a pull request on github: https://github.com/fsaxen/ParforProgMon

Adam Wyatt

Very nice. The progressbar currently doesn't work inside Live Scripts because of the way Live Scripts work - a *very* simple fix is to add "'Visible', 'on'" to the figure options when creating the figure (line 196 "progressbar.m"). It would be really good if the progressbar returned the handle to the figure and if that handle was accessible in the ParforProgressbar object - this would allow embedding into existing figures etc.

Liam Hall

Frerk Saxen

@Feng Cheng: Do you have an idea why pctconfig doesn't work for you? Can you create a more detailed bug report on github (https://github.com/fsaxen/ParforProgMon)?

Feng Cheng

"Error using icinterface/fopen (line 83)
Unsuccessful open: Unknown RemoteHost: ........."

Solved after i modified the o.ServerName
"% pct = pctconfig;
% o.ServerName = pct.hostname;
o.ServerName = char(getHostName(java.net.InetAddress.getLocalHost));
"

Hans Wurst

To answer my question, yes modify "progressbar.m" to
progfig = figure(...
'Units', 'normalized',...
'Position', [left bottom width height],...
'NumberTitle', 'off',...
'Resize', 'off',...
'MenuBar', 'none',...
'Visible', 'on');

Hans Wurst

I would like to open it in a live script, is it possible?

Frerk Saxen

@Emil Pavlov: Because this implementation uses progressbar (https://de.mathworks.com/matlabcentral/fileexchange/6922-progressbar) due to its very fast execution, adding elapsed time would require to change the progressbar implementation. Maybe ask for a feature request there?

Emil Pavlov

Great tool. Would you be able to add elapsed times as well?

Thomas Koopman

Ivan Mostak

Adel

Seems nice but unfortunately requires Instrument Control Toolbox !

Adel

Dylan Muir

Looks nice!

MATLAB Release Compatibility
Created with R2018b
Compatible with any release
Platform Compatibility
Windows macOS Linux

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!