Can I save the outputs of completed sims when running parsim instead of saving them all at the end?

I'm running 1000 simulations using parsim on 25 parallel workers. When I monitor my memory, I see that my memory usage goes up every time sims are completed, eventually reaching close to 100%. Fortunately, this has not caused my simulations to fail from a memory overload. However, I'd like to stay on the safer side, and save my sim outputs as soon as a batch is complete, and then clear that data from memory.

My code looks like this:

simOut = parsim(simInArray, ...
                'SetupFcn', @()utilities.foo(mdl, bar}), ...
                'CleanupFcn', @()close_system(mdl, 0), ...
                'ShowSimulationManager', 'on');

Where simInArray is 1000 elements long. Is there anyway I can dump every 25 runs to disk, and clear them from memory? Could there be a name-value pair into parsim that does this?

[Kadhir]

A follow-up question is, how does MATLAB continue to run simulations and keep their output in memory if memory usage is close to 100%?

[Walter Roberson]

Is there anyway I can dump every 25 runs to disk, and clear them from memory?

No, there is not.

Potentially you could save results using To File blocks.

Answer 1

[Paul]

Hi Kadhir,

Based on the description in the question, I'm not quite sure what's happening. How exactly are you monitoring the memory and correlating to when each simulation is complete? Is that through the Simulation Manager (I'm not familiar with that tool)?

One option would be to not run all 1000 simInArray objects in one go. You could, for example, do four groups of 250 and stitch the results together afterwards. I don't think this approach is too palatable, but thought I'd mention it.

If the issue is that the logged data that gets returned to simOut is too large, you can instead force all logging to .mat files. See this discussion for an example.

More generally, check out Work with Big Data for Simulations and Running Multiple Simulations and other Related Topics therefrom to see if anything resonates to address your concern.

[Kadhir]

I think this batching approach would work for our needs. Thank you!