I want to know how to understand the sampling time ts of the agent in reinforcement learning, and how its value will affect the training and results?

I want to know how to understand the sampling time ts of the agent in reinforcement learning, and how its value will affect the training and results?

Answer 1

[Emmanouil Tzorakoleftherakis]

Think of it as how often you need the agent to make decisions. Sometimes you need to make decisions very frequently (e.g. if your system dynamics are fast) and sometimes not. Could be viewed as the equivalent of a controller rate.

The smaller ts is, the more frequently the agent will make decisions, but that will make training much slower. On the other hand, if ts is large, training may progress faster, but depending on your system it may not converge.

Hope this helps

[嘻嘻]

@Emmanouil Tzorakoleftherakis

That's true. If I turn ts up, the training says "there's a singularity," which I can interpret to mean that if the frequency is too small, I can't control the system. I understand your answer and thank you very much.

[嘻嘻]

@Emmanouil Tzorakoleftherakis

My controlled system is a nonlinear coupled system with three inputs and three outputs, I want to use three PID controllers to control these three channels respectively, can I use only one agent, the nine action values of the agent are nine control parameters, the observation is the error and output value of each channel, the reward function is the error weighted sum of each channel. I'm not sure if this plan is reasonable, and I'd like your advice. Thank you very much.

[Emmanouil Tzorakoleftherakis]

You should be able to use a single agent for this application. However, since you already have a model of the system, I would first try with a model-based controller like MPC.