Parfor loop with complex structure

My idea is to run multiple independent simulations only changing parameters, but I want to keep all the results stored in 1 unique workspace that is also loaded so when I stop and rerun the code it checks from which iteration it should continue.

For this purpose I have 3 nested loops, the inner one being a parfor loop and the outer ones a for loop. The problem is that for all this data to be organized I want to dynamically create sublevels on the workspace structure named after the iteration indexes and I want to store 12 different variables with such different dimensions and types.

I tried to index the first level of the output structure as in examples, also creating an empty structure inside the loop but it does not work. Also, it seems save() cannot be executed inside a parfor loop.

Is there some workaround or method to accomplish my idea?

version_of_mpc='v5_2';

for days_horizon=1:7
    days_horizon

    for minutes_samplingtime=3:6
        minutes_samplingtime

        parfor cost_of_benefit_index=1:30
            
            cost_of_benefit=cost_of_benefit_index*10;
            outputfile_name1=['results_Hp' num2str(days_horizon,'%.2f') 'days_Ts' num2str(minutes_samplingtime,'%.2f') 'min_weightCost' num2str(cost_of_benefit,'%.2f') '_FEAS.fig'];
            outputfile_name2=['results_Hp' num2str(days_horizon,'%.2f') 'days_Ts' num2str(minutes_samplingtime,'%.2f') 'min_weightCost' num2str(cost_of_benefit,'%.2f') '_INFEAS.fig'];
            outputfile_name3='results.mat';

            full_name1=fullfile(['figures directory...' version_of_mpc],outputfile_name1);
            full_name2=fullfile(['same figures directory...' version_of_mpc],outputfile_name2);
            full_name3=fullfile(['Workspace output file directory...' version_of_mpc],outputfile_name3);

            cost_of_benefit

            if not (isfile(full_name1) || isfile(full_name2)) % checking what simulations are done

                [Xhist, Uhist_corrected, Uhist_controller, cost_prod_hist, pgrid_hist, i, percent, irradiation_plt, P_pv_plt, Price_energy_plt, CPU_time, total_simulation_time]=mpc_sim(minutes_samplingtime,days_horizon,cost_of_benefit);            
                
                plot_and_save_v4(Xhist, Uhist_corrected, Uhist_controller, cost_prod_hist, pgrid_hist, i, percent, irradiation_plt, P_pv_plt, Price_energy_plt, CPU_time, total_simulation_time,minutes_samplingtime,days_horizon,cost_of_benefit,version_of_mpc,full_name3);

                first_lvl_name='FEAS'
                
                if percent < 100
                    first_lvl_name='INFEAS'
                end

                results.(version_of_mpc).('ALL').(['Hp_' num2str(days_horizon,'%.0f') 'days']).(['Ts_' num2str(minutes_samplingtime,'%.0f') 'min']).(['weightCost_' num2str(cost_of_benefit,'%.0f')]).('Xhist')=Xhist;
                results.(version_of_mpc).('ALL').(['Hp_' num2str(days_horizon,'%.0f') 'days']).(['Ts_' num2str(minutes_samplingtime,'%.0f') 'min']).(['weightCost_' num2str(cost_of_benefit,'%.0f')]).('Uhist_corrected')=Uhist_corrected;
                results.(version_of_mpc).('ALL').(['Hp_' num2str(days_horizon,'%.0f') 'days']).(['Ts_' num2str(minutes_samplingtime,'%.0f') 'min']).(['weightCost_' num2str(cost_of_benefit,'%.0f')]).('Uhist_controller')=Uhist_controller;
                results.(version_of_mpc).('ALL').(['Hp_' num2str(days_horizon,'%.0f') 'days']).(['Ts_' num2str(minutes_samplingtime,'%.0f') 'min']).(['weightCost_' num2str(cost_of_benefit,'%.0f')]).('cost_prod_hist')=cost_prod_hist;
                results.(version_of_mpc).('ALL').(['Hp_' num2str(days_horizon,'%.0f') 'days']).(['Ts_' num2str(minutes_samplingtime,'%.0f') 'min']).(['weightCost_' num2str(cost_of_benefit,'%.0f')]).('pgrid_hist')=pgrid_hist;
                results.(version_of_mpc).('ALL').(['Hp_' num2str(days_horizon,'%.0f') 'days']).(['Ts_' num2str(minutes_samplingtime,'%.0f') 'min']).(['weightCost_' num2str(cost_of_benefit,'%.0f')]).('i')=i;
                results.(version_of_mpc).('ALL').(['Hp_' num2str(days_horizon,'%.0f') 'days']).(['Ts_' num2str(minutes_samplingtime,'%.0f') 'min']).(['weightCost_' num2str(cost_of_benefit,'%.0f')]).('percent')=percent;
                results.(version_of_mpc).('ALL').(['Hp_' num2str(days_horizon,'%.0f') 'days']).(['Ts_' num2str(minutes_samplingtime,'%.0f') 'min']).(['weightCost_' num2str(cost_of_benefit,'%.0f')]).('irradiation_plt')=irradiation_plt;
                results.(version_of_mpc).('ALL').(['Hp_' num2str(days_horizon,'%.0f') 'days']).(['Ts_' num2str(minutes_samplingtime,'%.0f') 'min']).(['weightCost_' num2str(cost_of_benefit,'%.0f')]).('P_pv_plt')=P_pv_plt;
                results.(version_of_mpc).('ALL').(['Hp_' num2str(days_horizon,'%.0f') 'days']).(['Ts_' num2str(minutes_samplingtime,'%.0f') 'min']).(['weightCost_' num2str(cost_of_benefit,'%.0f')]).('Price_energy_plt')=Price_energy_plt;
                results.(version_of_mpc).('ALL').(['Hp_' num2str(days_horizon,'%.0f') 'days']).(['Ts_' num2str(minutes_samplingtime,'%.0f') 'min']).(['weightCost_' num2str(cost_of_benefit,'%.0f')]).('CPU_time')=CPU_time;
                results.(version_of_mpc).('ALL').(['Hp_' num2str(days_horizon,'%.0f') 'days']).(['Ts_' num2str(minutes_samplingtime,'%.0f') 'min']).(['weightCost_' num2str(cost_of_benefit,'%.0f')]).('total_simulation_time')=total_simulation_time;
            
                results.(version_of_mpc).(first_lvl_name).(['Hp_' num2str(days_horizon,'%.0f') 'days']).(['Ts_' num2str(minutes_samplingtime,'%.0f') 'min']).(['weightCost_' num2str(cost_of_benefit,'%.0f')]).('Xhist')=Xhist;
                results.(version_of_mpc).(first_lvl_name).(['Hp_' num2str(days_horizon,'%.0f') 'days']).(['Ts_' num2str(minutes_samplingtime,'%.0f') 'min']).(['weightCost_' num2str(cost_of_benefit,'%.0f')]).('Uhist_corrected')=Uhist_corrected;
                results.(version_of_mpc).(first_lvl_name).(['Hp_' num2str(days_horizon,'%.0f') 'days']).(['Ts_' num2str(minutes_samplingtime,'%.0f') 'min']).(['weightCost_' num2str(cost_of_benefit,'%.0f')]).('Uhist_controller')=Uhist_controller;
                results.(version_of_mpc).(first_lvl_name).(['Hp_' num2str(days_horizon,'%.0f') 'days']).(['Ts_' num2str(minutes_samplingtime,'%.0f') 'min']).(['weightCost_' num2str(cost_of_benefit,'%.0f')]).('cost_prod_hist')=cost_prod_hist;
                results.(version_of_mpc).(first_lvl_name).(['Hp_' num2str(days_horizon,'%.0f') 'days']).(['Ts_' num2str(minutes_samplingtime,'%.0f') 'min']).(['weightCost_' num2str(cost_of_benefit,'%.0f')]).('pgrid_hist')=pgrid_hist;
                results.(version_of_mpc).(first_lvl_name).(['Hp_' num2str(days_horizon,'%.0f') 'days']).(['Ts_' num2str(minutes_samplingtime,'%.0f') 'min']).(['weightCost_' num2str(cost_of_benefit,'%.0f')]).('i')=i;
                results.(version_of_mpc).(first_lvl_name).(['Hp_' num2str(days_horizon,'%.0f') 'days']).(['Ts_' num2str(minutes_samplingtime,'%.0f') 'min']).(['weightCost_' num2str(cost_of_benefit,'%.0f')]).('percent')=percent;
                results.(version_of_mpc).(first_lvl_name).(['Hp_' num2str(days_horizon,'%.0f') 'days']).(['Ts_' num2str(minutes_samplingtime,'%.0f') 'min']).(['weightCost_' num2str(cost_of_benefit,'%.0f')]).('irradiation_plt')=irradiation_plt;
                results.(version_of_mpc).(first_lvl_name).(['Hp_' num2str(days_horizon,'%.0f') 'days']).(['Ts_' num2str(minutes_samplingtime,'%.0f') 'min']).(['weightCost_' num2str(cost_of_benefit,'%.0f')]).('P_pv_plt')=P_pv_plt;
                results.(version_of_mpc).(first_lvl_name).(['Hp_' num2str(days_horizon,'%.0f') 'days']).(['Ts_' num2str(minutes_samplingtime,'%.0f') 'min']).(['weightCost_' num2str(cost_of_benefit,'%.0f')]).('Price_energy_plt')=Price_energy_plt;
                results.(version_of_mpc).(first_lvl_name).(['Hp_' num2str(days_horizon,'%.0f') 'days']).(['Ts_' num2str(minutes_samplingtime,'%.0f') 'min']).(['weightCost_' num2str(cost_of_benefit,'%.0f')]).('CPU_time')=CPU_time;
                results.(version_of_mpc).(first_lvl_name).(['Hp_' num2str(days_horizon,'%.0f') 'days']).(['Ts_' num2str(minutes_samplingtime,'%.0f') 'min']).(['weightCost_' num2str(cost_of_benefit,'%.0f')]).('total_simulation_time')=total_simulation_time;    
            
            % i don't save any .mat file, i don't know how.
            % i even tried evalin inside of the plot_and_save function in
            % wich saving the figure works, but neither the creation of the
            % structure nor saving the .mat file worked
            end
        end
    end
end

%system('shutdown -s');

[Edric Ellis]

This code is not executable, so it's not exactly clear what you're trying to do here. Could you please try to narrow your problem down to a simpler example that still demonstrates the problem you're trying to solve.

One other separate thought is that your nested struct may not be the best way to store your results in any case. You might be better off storing things in a table. (The hierarchical structure might not be so easy to see using that method, but it would be much simpler to programatically access groups of data).

[Pol Cardona Rubio]

Hello @Edric Ellis, thanks for your response and recommendation about using tables. I will look in to it.

I wanted to store in 1 unique file the temporary data of all independent parfor loop iterations. This data should be dynamically named and structured for easy indexing afterwords.

For now, the workaround I've found is to save each parfor iteration data in separate dynamically named .mat files without any structure, all variables directly stored in the first level. Then outside the parfor loop I concatenate all independent .mat files into one.

I've tried creating only 1 .mat file and keeping it "writable" with the matfile() function, which would be the perfect solution and apparently worked at first, but I think that for some delays on reading/writing data simultaneously using parfor loops... the matfile ends up being corrupted, at least in case I force stop the simulation. I get the same error as: Error using load: unable to read file - (mathworks.com)

I think it would be a good feature to have a more straight-forward way of storing data, dynamically created, in one unique file inside a parfoor loop.

[Jan]

@Pol Cardona Rubio: There cannot be a reliable method to write to one file from multiple threads.

[Pol Cardona Rubio]

Hello @Jan, I've learned a lot from your solutions.

I'm not expierienced at all, but based on the problems I've encountered... merging structured .mat files with different field names it's quite messy, and the results are not pretty/flexible and I've seen a lot of questions around this topic.

Some kind of FIFO pipe for the order of access and saving data of the respective threads could not be a reliable method? Just out of curiosity.

Thanks.

[Jan]

@Pol Cardona Rubio: You can find some semaphore implementations in the FileExchange: https://www.mathworks.com/matlabcentral/fileexchange?q=semaphore . If only one thread has writing access to the file at a specific time, this should keep the file clean. As far s I remember, these semaphores are working on a single machine also, so if the parallel pool spans multiple computers, this will not work.

[Pol Cardona Rubio]

Uau!! Thanks @Jan, I'll look in to it!

[Pol Cardona Rubio]

For anyone interested. The best approach I found that works very well with parallel simulations is to create an exportAndPlot(results) function that is called at each independent iteration of the parfor loop. Nevertheless, the path or the name of the .mat and figures/pdfs/etc exported is a variable that is assigned at each iteration of the parfor (with some specific name code for you to understand later, but strictly different and unique at each iteration), this variable is passed to exportAndPlot(results,specificName). where you organize each simulation data and figures as needed and export results with the specific naming. This way you are able to generate in a parallel manner files with independent and custom organization of results inside a parfor loop. Then when the parfor loop is finalized you just go through all .mat independent test results and unify them in one unique .mat file to be able to analyze easily. When unifying, add columns to the results table that respresent the specific set of parameters of each test to always keep track of the specific test setup. Also, to not repeat already finalized test runs if the overall tests crashes or for whatever other reason you have to repeat the main.m execution, only run the simulation.m file inside the parfor loop if and only if the corresponding .mat results file at the specific path does not exist yet, this is implemented with an if statement. This methodology could be a very useful to be integrated by Matlab, where a simulation.m file, an exportAndPlot.m, a customNaming.m and customPath.m files are selected and hence complex simulations can be executed in parallel easily with modularity and custom results output organization.