Performance of applications that are developed in MATLAB versus C++/C

Hello,

I have submitted a journal paper that compares the performance of two different implementations of power flow algorithm on multi-core CPUs: one developed in C language with parallelization based on OpenMP and Intel MKL libraries, and another one developed in MATLAB and based on vectorization techniques. Performance is comparable for small to medium size systems while the C implementation is up to 2.5 times faster than MATLAB for large systems, which makes sense!

However, one of the journal reviewrs is saying that the scripts based on Fortran and C++ language are usually running faster than any other computational languages, and according to the reference [1] shown as follows, the MATLAB is usually 9 to 11 times slower than the best C++ executable.

[1] http://jonathankinlay.com/2018/10/comparison-programming-languages/

I believe that this comparison does not reveal the full performance of MATLAB. To address this reviewer's concern, I need the following information:

1. Is there a performance comparison study between MATLAB and other programming languges published by MathWorks?
2. Can you name the external high performance libaries that are used by MATLAB (e.g., vectorization, linear system solvers, matrix operations)?
3. Do you have any comment/suggestion that can help in addressing this concern?

Thank you very much in advance.

Afshin

Answer 1

[James Tursa]

Any such comparison will depend on exactly what computations are in question. A general comparison of languages probably isn't going to be meaningful. For example, linear algebra (e.g., matrix multiply) in MATLAB is done by calling 3rd party BLAS libraries that are highly optimized for speed using multi-threading and memory access techniques that optimize cache usage etc. There is no way you would be able to write any C/C++ routine by hand, even using OpenMP, that could rival this for speed.

[Afshin Ahmadi]

I totally agree with you! I need to collect further information to provide a convincing response to the reviewer.

[James Tursa]

1. I am unaware of any such studies
2. The BLAS and LAPACK libraries would be one example
3. I am not sure what is the "concern"

The link you provide simply states conclusions based on the coding of one particular algorithm. The speed would depend on how it was coded and MATLAB version involved. Unfortunately, the original article link is broken, so there is no way for us to examine the MATLAB coding techniques used and hence gives us nothing to comment on. But at "9 to 11 times slower", I can tell you it wasn't dominated by linear algebra calls in MATLAB.

[Afshin Ahmadi]

Thank you James. I found the article in this link: https://www.nber.org/papers/w20263

[James Tursa]

Looks like the MATLAB code used is a simple for-loop with a short-circuit break. The authors do mention vectorized MATLAB code, but at the same time admit that the algorithm they chose for comparison purposes is not very good for the types of vectorization that MATLAB can provide for other algorithms (presumably because of the short-circuit potential). So you can restate their observation as "MATLAB using a for-loop is 9 to 11 times slower than C++".

So yeah, if you pick an algorithm that is not good at vectorization within what MATLAB provides, it is not surprising that it is 9 to 11 times slower. I'm surprised the number isn't even higher than that, quite frankly.

[Afshin Ahmadi]

Thanks so much for your insights. I appreciate you taking the time to help me.

Answer 2

[Image Analyst]

If you use MATLAB you can use more powerful algorithms that are optimized for speed and verified for no bugs. Imagine if you had to write and validate your own code for regionprops() or bwconncomp()!! It would take a very long time and pure speed tests on simple benchmarks don't take that into account. So cherry picking situations a simple situation where MATLAB is slower (like using a loop when it could be vectorized) is not a fair comparison in my opinion.

My programs are typically 4000 to 7000 lines of code, with extensive use of built-in functions. If I had to write all those functions myself, or even use open source functions, it would take much longer and that development time is not taken into account in a pure speed test done after all code writing has been finished. I don't think that development time should be discounted/ignored.

[Afshin Ahmadi]

I totally agree with you.