[Optimization Tool] Parallelization not working

I have a problem using the Optimization Tool with parallel evaluations of the fitness functions.

My fitness function starts an external program using the 'system(external program.exe -par1 - par2 .....)'. This external program calculates the fitness and write the fitness value in a text file. The fitness function reads this value and returns it to Matlab. Everything is working smoothely when i use the option "in serial" at "evaluate fitness and constraint functions".

Now I want to run the fitness program several times parallel. So I use >> matlabpool open local 4 and switch "Evaluate fitness and constraint functions" to "in parallel".

After I press start, I can see in the task manager that my external programm is getting started 4 times but only for 2 seconds. Then it gets killed. There are no error messages nor a window of the external program.

My guess is that Matlab is opening the external program in a different way. Any idea to avoid this or to solve this problem

Thanks! Andi

[Sean de Wolski]

Can you manually run the external program 4x simultaneiously?

[Andreas]

Sorry, i just see your answer: yes, i can

Answer 1

[Titus Edelhofer]

Hi,

I suspect the problem on the external program side: you might try to start manually the external program twice. E.g. can you tell your external program to use a different output file name for each loop? I.e., something like

outFile = ['result_' num2str(i)];

because otherwise all 4 instances of the program try to write to the same file (which will fail) ...

Titus

[Andreas]

Hi! Thanks for your answer.

The external program reads the data from different files and produce different files. So that is not the problem.

I also tried to start the fitnessfunctions with "pause(randi(20))". So the external programm will not start at the same time. But still not working.

Answer 2

[Jason Ross]

If you get the return values of system(), is there anything interesting in there?

[status, result] = system('command')

[Andreas]

Unfortunately not:

status =

     0

result =

     ''

Answer 3

[Alan Weiss]

The documentation states that accessing external files is problematic when using parallel computation.

I suppose that your fitness function computation is by far the costliest part of the calculation, and yet it seems you have only one computer performing the fitness calculation. So I do not see how you expect parallel computation to be beneficial. If each core could send its computation to a separate computer, and if you could get the results without writing files, then you might get speedup by computing in parallel. As it is, I am afraid that you are not going to see speedup, even if you could get the parallel computation to work.

And let me say one more thing. It seems that you are performing a genetic algorithm optimization, since you call your objective function a "fitness function." I believe you will have faster, more reliable results by using patternsearch instead of ga. The only difference in syntax is ga does not require a starting point, but patternsearch does. If you have finite lower and upper bound vectors lb and ub, you can get random initial points for patternsearch as follows:

 x0 = lb + rand(size(lb)).*(ub - lb);

Good luck,

Alan Weiss

MATLAB mathematical toolbox documentation

[Andreas]

Yes, the fitness function is really by far the costliest part of the calculation (around 1 hour). The external programm is very expensive, so we have only a license to use on one CPU core BUT i am allowed to use the program several times. The server has 8 cores, so I can calculate max. 8 fitess functions parallel with of course a speedup of max 8.

Thanks Alan for this hint. Patternsearch sounds great bcz I have already a starting point. But first I need to be able to start the program...

Thanks!

[Jason Ross]

The plot thickens :)

I assume that there is a license manager of some sort running that the software communicates with, and that the license manager has a log file of the licenses being checked in and checked out. Can you get access to that log and see if it shows any errors about license counts being exceeded, etc?