Using Parallel computing with external software

Hi. Im trying to use Parallel computing with a function that calls an external software (Ansys). Im using optimization with Genetic Algorithm. But i keep receiving an error. I start the pool with 3 cores.

main code:

clc; clear;
termino=1;                             % just a parameter
funcao = @(x) Eval_v1_3(x,termino);    % objective function with 3 variables
LB = [0.7 1.0 0.5];                    % lower bounds
UB = [3.0 3.0 0.7];                    % upper bounds
options = optimoptions('ga','MaxGenerations',20,'PopulationSize',10,'UseParallel',true);  % sets PARALLEL
x = ga(funcao,3,[],[],[],[],LB,UB,[],options);

objective function:

function val = Eval_v1_3(x,termino)
dlmwrite('param_otim.txt',[x(1) x(2) x(3) termino],'precision','%.5f');           % writes the individual to be read in the middle of Ansys execution
!C:\"Program Files"\"ANSYS Inc"\v150\ansys\bin\winx64\ANSYS150 -b -i "code.txt" -o "output.txt" -np 1  % call Ansys and execute an algorithm that reads param_otim.txt
T_A=load('temps_monit_A.txt','-ascii'); T_B=load('temps_monit_B.txt','-ascii');    % reads the values generated by the simulation
rp = 1;  t_sol1 = 1467; t_sol2 = 840;                                              % just parameters
val = rp*sqrt(((t_sol1 -T_A(1))^2 + (t_sol1 -T_A(2))^2 + (t_sol2-T_B(1))^2 + (t_sol2-T_B(2))^2 )); % objective function 
dlmwrite('monitoramento.txt',[x T_A T_B -val],'delimiter','\t','-append');         % writes every individual and result for monitoring purpose
end

error that appears:

The process cannot access the file because it is being used by another process. 
The process cannot access the file because it is being used by another process. 
The process cannot access the file because it is being used by another process.

Can i discover which file could not be access? The problem is in Ansys or Matlab? Is there a better way to code the problem (still using GA)?

It works without the parallel, but i would like to speed up with parallel computing.

Thanks!

Answer 1

[Mechrod]

This week i came back to work with this again, and after some research and tests, this is the answer for the main file:

clc; clear;

spmd
  mkdir(sprintf('worker%d', labindex));
  copyfile('file1.db',sprintf('worker%d/',labindex));
  copyfile('file2.mp',sprintf('worker%d/',labindex));
  copyfile('file3.txt',sprintf('worker%d/',labindex));
  cd(sprintf('worker%d', labindex));
end

 funcao = @(x) ZF_Eval_v1_3_new(x,termino);
 LB = [0.7 1.0 0.5];           
 UB = [3.0 3.0 0.7];            
 options = optimoptions('ga','MaxGenerations',20,'PopulationSize',10,'UseParallel',true);  % sets  PARALLEL
 x = ga(funcao,3,[],[],[],[],LB,UB,[],options);

and the function file remains the same:

 function val = ZF_Eval_v1_3_new(x,termino)

 dlmwrite('param_otim.txt',[x(1) x(2) x(3) termino],'precision','%.5f');           % writes the individual to be read in the middle of Ansys execution
 !C:\"Program Files"\"ANSYS Inc"\v150\ansys\bin\winx64\ANSYS150 -b -i "file3.txt" -o "output.txt" -np 1
 T_A=load('temps_monit_ZF.txt','-ascii'); 
 rp = 1;  t_sol1 = 1467; t_sol2 = 840;                                              % just parameters
 val = rp*sqrt(((t_sol1 -T_A(1))^2 + (t_sol1 -T_A(2))^2 + (t_sol2-T_A(3))^2 + (t_sol2-T_A(4))^2 )); % objective function 
 end

The execution time was reduced in 40%. As Walter said, the key was to mkdir and copy the files to the workers folders. I used 3 workers for testing, so it created 3 folders, copying all the files i needed on each folder.

Hope its usefull for someone!

[Min Zhang]

Hello, I am having a very similar issue. I am wondering whether you could have a look at my post here.

https://www.mathworks.com/matlabcentral/answers/435658-issues-with-running-genetic-algorithm-ga-in-parallel-mode

Thanks in advance!

Answer 2

[Walter Roberson]

You need to use a different input and output file names for the different workers. With the code you are using now, all of the workers are trying to write to the same file at the same time.

[Mechrod]

How can i use different file names and assign then to each work (core), without "confusing" Matlab?

[Walter Roberson]

I would suggest using tempname() to create the input and output file names.

It looks to me as if it would be necessary to create a version of code.txt that referenced the input file name.

However, another approach would be to use a different directory for each worker, copying in the appropriate input files and executing.

Your line

dlmwrite('monitoramento.txt',[x T_A T_B -val],'delimiter','\t','-append');         % writes every individual and result for monitoring purpose

is going to be a potential problem. Are those values being written scalars or vectors? If you must write to a common output file, it is possible to do it without accidentally overwriting something that was just written, but I am not certain whether Windows would decide the attempts were clashing... it might.

The safe way of writing involves using fopen() of the file with permission of 'a+', sprintf() of all of the output into a string, and then fwrite() of the string into the file as a single operation, after which you fclose() the file. In those circumstances, each fwrite() would be certain of being done "atomically" without interference from the other processes.

[Mechrod]

Hi. Sorry for my stupidity, but how can i use different folders?? It seems to be the easiest way. How can i control whats going to each worker?

Regarding my "monitoring" line, i can suppress that and solve this later.

I also tried using this:

options = optimoptions('ga','MaxGenerations',20,'PopulationSize',10,'UseVectorized',true);

and changed this line in the function:

dlmwrite('param_otim.txt',[x(:,1) x(:,2) x(:,3) termino],'precision','%.5f');

But i got this error:

Error using horzcat
Dimensions of matrices being concatenated are not consistent.

Error in Eval_v1_3 (line 6)
dlmwrite('param_otim.txt',[x(:,1) x(:,2) x(:,3) termino],'precision','%.5f');

Error in teste_ga>@(x)Eval_v1_3(x,termino)

Error in createAnonymousFcn>@(x)fcn(x,FcnArgs{:}) (line 11)
fcn_handle = @(x) fcn(x,FcnArgs{:});

Its possible to use Vectorize? Maybe a parfor? Maybe some use of smpd like this?

[Walter Roberson]

dlmwrite('param_otim.txt',[x(:,1) x(:,2) x(:,3) repmat(termino, size(x,1),1)],'precision','%.5f');

[Walter Roberson]

To use different directories for different workers, you could parfevalOnAll of code that does a tempname to get a unique name, then mkdir()'s it, copyfile()'s code.txt to there, and cd's to it. After that you can start your ga with UseParallel.

[Jason Xu]

Hi Walter, im having the same issue like Mechrod. Im wondering:

I got external software model, GA objective function, GA output function and main function(to run ga) saved in the same folder. The objective function includes the matlab code to open the model, modify the model input parameters,run the model simulation and read model results(which is a .mat file automatically generated in the same folder where the related model locates by the time model simulation is finished). Which files should i parfevalOnAll or copyfile to the relevent worker folders?(like Mechrod did in the accepted answer)

Thank you!

[Walter Roberson]

You are probably okay not copying the .m files to the worker.

With MS Windows, there is the possibility that any file used as input by the external software might be locked against reading and writing as long as the software has the file open.

With MS Windows, any file use as output by the external software would likely be locked against reading and writing as long as the software has the file open.

These are not always the case, but the interface for opening files automatically locks files unless the code specifically asks that the file not be locked by the default mechanism.

When a program knows enough to ask that files not be locked by the default mechanism, the program will typically ask for a portion of the file to be locked while the code is making changes to the file. This can lead to temporary denials of access for particular operations.

Any time an application enables shared reading access to a file it is writing to, you need to beware that the content you read just a moment ago might not be the same as the content that is there now.

Unix systems deal with the situation a bit differently: most locking on Unix systems is advisory rather than mandatory. Meaning that your program is expected to know about the possibility of locks and to deliberately ask about them. The situation is a bit more risky. On the other hand, is is very common for multiple programs to want to read from the same file, and on Unix that is no problem as long as nothing is writing to the file; whereas on MS Windows, the same case of multiple programs wanting to read from the same file will typically result in the file being reported as inaccessible to everything except the first process that opens it.

[Jason Xu]

Hi Walter:

After some time of testing and researching, i changed my parallel optimization GA objective function as follow: ( assume parpool(2) )

%% Objective function
function    y     = objective(x)
var.i1      = x(1);       % variable 1
var.i2      = x(2);       % variable 2
var.i3      = x(3);       % variable 3
i_worker    = get(getCurrentTask(),'ID');   % get the worker index
if i_worker == 1
    [y]      = subobjective1(var)    % sub-objective function 1 
end
if i_worker == 2
    [y]      = subobjective2(var)    % sub-objective function 2 
end
end

and for each sub-objective function:

%% Sub-objective function 1
function    y       = subobjective1(var)
% copy the software model 'Model.mot' and renamed as 'Model_copy1.mot' and saved in the same folder
file.path     = ('E:\...\')                                        % file path 
file.name     = dir(strcat(file.path,'Model.mot'));                % this is the original software model 'Model.mot'   
copyfile([file.path 'Model.mot'],[file.path 'Model_copy1.mot'])    % copy the software model and renamed as Model_copy1.mot
% build the connection between external software and matlab
mcad1         = actxserver('MotorCAD.AppAutomation');              % external software MotorCAD, using activeX
FileName      = strcat(file.path,'Model_copy1.mot');               % read the copied new model 'Model_copy1.mot'
res           = invoke(mcad1,'LoadFromFile',FileName);             % load the file
% changed the settings 
invoke(mcad1,'SetVariable','Magnet_Length',var.i1);                % just some model parameters changing                            
invoke(mcad1,'SetVariable','Rotor_Lam_Length',var.i2);             % just some model parameters changing 
invoke(mcad1,'SetVariable','Stator_Lam_Length',var.i3);            % just some model parameters changing 
% run the model
invoke(mcad1,'BuildModel_Lab');                                    % after running is finished, a same name folder 'Model_copy1' is generated in the same folder where Model_copy1.mot is located, 
                                                                   % also a result.mat file is generated within 'Model_copy1' folder which contains calculation result
% load the result file
y             =  load(file.path\Model_copy1\result.mat)
% quit the model
invoke(mcad1,'Quit');                                              % close the software model
end

%% Sub-objective function 2
% structurally the same as sub-objective function 1
% change 'Model_copy1.mot' to 'Model_copy2.mot'
% change 'mcad1' to 'mcad2'
% change load(file.path\Model_copy1\result.mat) to load(file.path\Model_copy2\result.mat)

Problem1: This strategy works fine only for the first two individuals, that is, after the two software models being closed and then re-opened, the mistake information shown on the software indicates that the same software files are being called (etc.Model_copy1 being called by both worker1 and worker2 ). I dont know why this happened since i have already made different assigments for different workers in my objective function. However when i open the same two software files (by mouse-clicking) and do the calculation then it works fine. Im really confused now.

Problem2: How to add the code to count the number of individuals which has been evaluated (fitness function) during one generation in the objective function?

Provlem3: I dont really understand the method of 'tempname', could u give me some examples? If i choose to add certain input files and execution files to the different workers dictionaries, which are the files should be added?

Thank you !

Update 5th June: make it clear about opening the model, calculation and read the result file.

[Walter Roberson]

I do not see the source code for the functions Model_copy1 or Model_copy2 ?

[Walter Roberson]

"Problem2: How to add the code to count the number of individuals which has been evaluated (fitness function) during one generation in the objective function?"

The plot function is called once per generation. Therefore you could share a variable in some manner, and increment the variable inside your objective function, and reset the variable in your plot function.

"Provlem3: I dont really understand the method of 'tempname', could u give me some examples?"

tempname() is guaranteed to give a different result each time it is called, even if it is called from different workers. So you can use code such as

%tempname should be unique so there should never be duplicates, but
%just-in-case
while true
  localdir = tempname;
  if ~exist(localdir, 'dir') && ~exist(localdir, 'file'); break; end
end
mkdir(localdir);
copyfile(fullfile(filepath, 'Model_copy1'), localdir)
y = Model_copy1_output(localdir);
rmdir(localdir, 's')

[Jason Xu]

Hi Walter:

i've edited my coding and now i think i made it clear about Model_copy1.mot and Model_copy2.mot. Could you have a look about my code again? Thank you!

[Walter Roberson]

load(file.path\Model_copy1\result.mat)

That involves two invocations of the mldivide (\) operator.

Nothing in your invocations appears to tell MotorCAD where to place the output.

[Jason Xu]

Hi Walter:

After the simulaton is run with the code below:

% run the model
invoke(mcad1,'BuildModel_Lab');  

and by the time simulation is finished, MotorCAD automatically generated a folder with the same name as the model (eg. Model_copy1), and within the folder Model_copy a file with fixed name result.mat is also generated by MotorCAD automatically. So all i need to do after the simulation is to locate the result.mat with respected to the relative model (eg. Model_copy1) and load this .mat file and then ill have my calculation output.

whats your suggestion for my scenario? Thank you!

Answer 3

[Jason Xu]

Hi Mechrod, nice work! Still I’m wondering what are those file1 file2 file3 that you have copied to your workers directories? In my understanding is it necessary to copy your ansys model and your calculation results text file as well? How do u make sure different ansys models are being called regarding to different workers? Thank you!

[Mechrod]

Hi Jason!

The file1 is the geometry+mesh file. Its always the same for all simulations, so every time Matlab evaluates a individual, it always use the same mesh.

File2 its just the material properties file, same considerations as above. I use like this because its easier to change the material used in my models.

File3 is my APDL algorithm. It implements my BC and IC, apply the correct loads and etc. This file reads the file generated in dlmwrite('param_otim.txt'...), that contains the parameters coming from a individual from the GA.

This is the workflow:

A - copy file1, file2 and file3 to all workers folders (this happens only one time);

B - for each function evaluation (each individual):

1 - use the command dlwrite to generate the individual in a form Ansys can read;

2 - run Ansys using "!C:\....." command. The 'code.txt' is the 'file3.txt' (sorry for using different names)(i edited my answer to correct this and other things);

3 - After running the simulation, i use that "load" command to get the file results generated by Ansys, and use this values to calculate my objective function "val".

So, in my function evaluation "function val = ZF_Eval_v1_3_new(x,termino)", Matlab enters an individual 'x', and retrieves the 'val' value generated by the results from this individual.

The step B and its sub-steps are all done inside one worker folder. Matlab takes charge on sending the next individual to a available worker.

Hope it helps!

[Jason Xu]

Hi Mechrod, thank you for your detailed answer. I think my model is working now! Still sometimes after several hundreds of interations when the matlab code ask the model to be saved and quit then the model just got frozen and crashed. So hows Ansys's stability? Did it happen to you ? Thank you!

[Mechrod]

Hi!

My Ansys works fine. When the individual has a bad combination of parameters, my simulation may not converge and simply exit. To avoid that, i found 2 different solutions:

1 - put all the objective function inside a if to check if the result file exists, like this:

if exist('results.txt', 'file') == 2 % File does exist

    %make the calculations
else
    val = 9999999;
end

If the results file does not exist, assign a very high value, to tell that this is a bad individual.

2 - In the beginning of your Ansys algorithm, write a dummy results file, that make a very bad result for the objective function. If the simulations converge, overwrite this file with the real results.

Hope it helps.