Parfor is running much slower on a 18-core PC than a 10-core PC

I am currently running a compute-intensive financial trading test; the test has 503 independent loops. I have two PCs in my office, one has 10-core Intel i9-10900k @ 3.7GHz with 128GB DDR4; the other one has 18-core Intel i9-7980XE @ 3.4GHz with 128GB DDR4 as well.

In order to speed up the whole processing, the first 10-core PC runs loop 1-to-189 in parfor; the second 18-core PC runs loop 190-to-503 in parfor.

After one-day running, I found the 10-core PC finished 25 loops, the 18-core PC only finished 4 loops. I found very curious, can anyone know the reason of this problem?

P.S. 10-core PC running MATLAB 2020b with latest updates with MOSEK 9.2.35

18-core PC running MATLAB 2021a with latest updates with MOSEK 9.2.36

I also checked the taskmgr.exe, unlike the 10-core PC, I found in 18-core PC, 10 matlab tasks are in one group, the other 10 tasks are seperated....

[Raymond Norris]

One suggestion is to run R2020b on the 18-core machine in order to compare the same versions of MATLAB. Or alternatively, run R2021a on the 10-core.

[Walter Roberson]

Also, it would be interesting to try restricting the parpool size to 10 on the 18 core machine.

[Ive J]

On a different note, those CPUs have different frequencies, so I'm not sure even trying the same MATLAB version would be an unbiased comparison. My personal experience with different solvers (including MOSEK, GUROBI and IBM-CPLEX) showed that core numbers doesn't necessiarily means faster calculations. You may need to optimize the number of workers on each machine seperately to increase the overall performance. Also, I guess MOSEK automatically solves in parallel, so you may also want to compare the performance of both machines under single core scenario.

[David Goodmanson]

From simple extrapolation it looks like if you invest in an even better processor with 20 or more cores, the calculation will grind to a complete halt.

[Walter Roberson]

parfor has communications overheads, and if you give each node too little to do, then you can end up spending more time communicating than you get back from splitting the work. So it is possible for progress to get slower as you add more workers. It would be interesting to restrict both machines to (say) four cores and see what their performance is.

The studies I looked at a number of years ago suggested that 4 to 6 cores was the "sweet spot" for a variety of tasks -- at least with the hardware of those days. There are tasks for which the communications might be comparatively small and so net performance can improve as you add cores, even into the tens of thousands, but most code has to be specially written for that kind of situation.

[Mingyang Sun]

When using the MOSEK (interior-point or integer programming) inside parfor loop, it is better to turn off the multi-threads in MOSEK, otherwise, by default, MOSEK will using all cores.

param.MSK_IPAR_NUM_THREADS = 1;

param.MSK_IPAR_INTPNT_MULTI_THREAD = 'MSK_OFF';

After I turn off the multi-threads in MOSEK, parfor-loop is now approx. 17 times faster than for-loop!