DDPG - Noise Model - sample time step - definition

Hello!

At the moment i am tuning the Parameters of my DDPG Algorithm and i don´t fully understand the Updating process of the Ornstein-Uhlenbeck Noise Model.

The Matlab Documentation describes the Process of Updating the Noise Model of a DDPG-algorithm, consisting of a Formula, which is used in every "sample time step".

(https://www.mathworks.com/help/reinforcement-learning/ref/rlddpgagentoptions.html Chapter: Input Arguments)

But how is the "sample time step" defined? Does this correlate with the episode or step count of the RL trainingprocess?

Thanks

Niklas

Answer 1

[Emmanouil Tzorakoleftherakis]

Hi Niklas,

This post should be helpful. By "sample time step" the documentation refers to the "step count of the RL trainingprocess", i.e. each episode consists of a number of time steps, and noise is applied to the selected action at the begining of each time step. See this link for a description of DDPG (and specifically step #1). The time step value can be specified in the agent options here.

[Niklas Reinisch]

Hello Emmanouil,

Thank you for your fast answer! It was helpful to understand the noise generation process.

But I got a consequential question, because the NoiseDecay(-Rate) is not explaned in the Link you mentioned. Am i right when saying:

?

I hope you can help me with that one as well.

Thanks

Niklas

[Emmanouil Tzorakoleftherakis]

Correct. This is why you want to keep the decay rate small or zero if you want to promote exploration.