Use current simulation data to initialize new simulation - RL training

In the context of PPO Agent training, I would like to use Welford algorithm to calculate the runninig average & and standard deviation of my observations, in order to standardize them and improve the convergence of actor & critic neural networks.

I implemented the algorithm, but I don't know how to keep track of the current running statistics (average and standard deviation) every time a new simulation starts, during the training. This is what I would like to do:

• Whenever a simulation terminates (i.e. "isDone" flag is set to 1) , save the current value of runnig statistics in Matlab workspace
• While initializing the new simulation, set the starting value of the running statistics to match the values just saved in Matlab workspace

Note that I'm using the standard "train" function to run the training, so the transition between one simulation and the next one is handled automatically and I don't have much flexibility in this sense.

I thought about using the "ResetFcn" function handle within my "SimulinkEnvWithAgent" object to accomplish the task, but I am still not able to programmatically save the last value of my signal to the Workspace at the end of a simulation, and then pass it to the ResetFcn as additional argument in order to initialize the next one

Answer 1

[Poorna]

Hi Federico Toso,

I see you want to save simulation data to workspace to later use it in your "ResetFcn". A suitable tool for this is the "rlDataLogger" object, which enables you to log simulation data at various points, such as after each step, episode and after each learn subroutine. You can craft a custom function for logging the specific statistics you're interested in and then assign this function to the appropriate callback property of the rlDataLogger. Although logging typically saves data to a folder after training concludes, your custom callback function can be used to immediately write the necessary statistics to the MATLAB workspace.

You can create a "rlDataLogger" object as below:

logger = rlDataLogger();

For instance, to log the ActorLoss value after every episode, your episode finish callback function could be structured like this:

function dataToLog = episodeFinish(data)
    assignin('base', 'actorLoss', data.ActorLoss);
    dataToLog = data.ActorLoss;
end

And then assign the function handle to the corresponding callback property of the data logger object as below:

logger.EpisodeFinishedFcn    = @episodeFinish;

To learn more about the "rlDataLogger" function refer to the below documentation:

https://www.mathworks.com/help/reinforcement-learning/ref/rl.logging.filelogger.html

Hope this Helps!

[Federico Toso]

Thank you!