Did parfor actually run on the cluster

Hi,

I was running a matlab code on a cluster with parfor to speed up my code, but I was wondering if parfor is actually run. So the piece of code that use parfor is the following

parfor ii = 1 : size( stateind , 1 )

disp( ii ) ;

Ibra = stateind( ii , : ) ; % Ordered set of indices for the bra state

for jj = ii : size( stateind , 1 )

I will print out ii for each parfor. In my local computer, I will get some random output as parfor randomly assign the loop to different workers. However, I got the following serial output on the cluster

Starting parallel pool (parpool) using the 'local' profile ...

63504

63503

63502

63501

63500

63499

63498

63497

63496

63495

63494

63493

63492

63491

63490

63489

63488

As a result, I am confused if parfor is actually run on the cluster or not. By the way, I didn’t specify the profile of the parpool, how should I properly do that?

UPDATE: When I replace parfor with for, indeed, the code seems run slower on the cluster. However, I still fell something is not correct here...

Thanks!

Answer 1

[Raymond Norris]

Hi Mao,

If a parpool is not already running, when MATLAB executes a parallel construct (e.g. parfor), then it will start one, using the default profile (usually local) and a default size (which is min(maxNumCompThreads,12) ). To override the the default profile/size, see the parpool command.

It doesn't appear as though the parpool is running for two reasons

• After starting the pool, you should also see (size, 2 in this case, will be specific to your environment)

Connected to the parallel pool (number of workers: 2).

• When a parpool doesn't start properly, it will run a parfor backwards

Raymond

[Mao Lin]

Exactly, I saw the for loop is running backwards! Do you have any idea how to fix it?

Thanks!

[Raymond Norris]

When the parpool starts, if memory serves, it needs to send files and data to the workers (even before calling parfor, spmd, etc.). In doing so, too much might have been sent and MATLAB decided to just run locally (admittedly, it would be better to notify the user first -- although perhaps that's why MATLAB doesn't display "Connected to ... ").

I'd try a couple of things

• We don't need to send anything from the MATLAB cached path, because that'll be on the worker path, but you might have a lot of custom folders. Consider removing what you can. Or when calling parpool, set AutoAddClientPath to false. This will prevent large, unnecessary, local files from being transferred.
• If you're running a MATLAB script, all data in the workspace needs to be transferred. Re-write your script as a function and pass arguments in and out as needed. This will minimize unnecessary data transfer.
• Rather than a local pool, try a threads pool (https://www.mathworks.com/help/parallel-computing/choose-between-thread-based-and-process-based-environments.html)

Raymond