mpi affecting matlab execution speed

I am running matlab on a nice cluster that has 16 nodes, each with dual octocore Xeon E5-2670s.

I have noticed that some of my code runs a lot faster when the node is being used by a colleague who is running OpenMP processes. Does this make any sense?

The code in question spends most of its time in a mex function, which launches 2 or 3 threads each time it is called. When I run a piece of test code, the mex function is called 500 times.

This test code completes in about 0.3 seconds when OpenMP is being used, but if I run on another node that does not have any openMP processes, the same code takes 0.8 seconds.

Normally, code runs slower when the machine is running other stuff at the same time, but here it seems to run faster!

My conclusion is that the overhead of launching a pthread is somehow reduced, but I have no idea why this would be.

Anyone have some insight?

[Jed]

I found that this is not really a matlab issue at all... the code linked here that creates 50000 threads that do nothing also runs much slower on the unloaded nodes.

(I used the code on the bottom half of linked page)

Answer 1

[Philip Borghesani]

Could the problem be a power savings setting? Many machines are now configured to run at a lower clock rate when not heavily loaded.

[Jed]

Good question... I was also wondering about that... perhaps I will google around and see if I can find out anything like that... but I still suspect it has something to do with openmp. perhaps I can verify this by running a different computationally intensive program that doesn't use openmp on one of the free nodes.

[Jed]

OK... it seems you are correct. I created a process that spawns 12 threads and just adds numbers like crazy. When I run this program, the performance of my thread launcher improves by a factor of >3x. Thanks.