I have some questions about the minibatchsize attribute of PPO+LSTM

Hello everyone

I found this sentence when I searched the content of PPO+LSTM in the official MATLAB documentation：“For a PPO agent, the trajectory length is the MiniBatchSize property of its options object” ，But I have some doubts about this statement。When using PPO+LSTM, does the agent no longer need to sample minbatch from the current experience sequence? How to understand "For a PPO agent, the trajectory length is the MiniBatchSize property of its options object".

Hope to get your answer

Answer 1

[Aneela]

Hi xiang,

“MiniBatchSize” refers to the number of samples from experience replay buffer that are used for one iteration of training.

• For non-recurrent neural networks, these samples are selected randomly because the network treats each input independently.
• LSTMs require a sequence of experiences to effectively learn temporal features.
• When using PPO with LSTM, the agent emphasizes on managing sequences of experiences to leverage the LSTM's ability to learn from temporally dependent data.

“For a PPO agent, the trajectory length is the “MiniBatchSize” property of its options object”:

• A trajectory is a sequence of states, actions, and rewards that an agent experiences in the environment from the start of an episode until a terminal state.
• The agent learns from trajectories of experiences.
• The “MiniBatchSize” value specifies the length of these trajectories.
• If “MiniBatchSize” is set to 50, the LSTM network will be trained on trajectories of experiences where each trajectory is 50 steps long.

For more information on PPO Agent options, please refer to the following MathWorks documentation: https://www.mathworks.com/help/reinforcement-learning/ref/rl.option.rlppoagentoptions.html