How does the type of solver (fixed vs variable step) affect reinforcement learning in Simulink environment?

So I successfully trained a robot manipulator to grab a object with a TD3 agent with the variable solver ode45. But when I changed the solver to a fixed solver (backward euler) with step size 0.01 (I also used rate transition block between the agent and the state,reward,done signal but the block read no operation since the agent and fixed solver was both at sample time = 0.01 so I'm assuming it had no affect) my agent performed worse and did not learn.

The reason why I used the rate transition block and changed to a fixed solver is because I'm trying to use a custom RL agent that I programmed on my own but I can't seem to use a variable solver otherwise the model doesn't compile when I try to train it. But it seems that fixed solver has negative effects on training.

Has anyone successfully trained a custom agent with a fixed solver or had similar issues? Any adivce or insight is greatly appreicated

[Torsten]

Try a smaller stepsize.

[Vincent]

I'll to reduce it and see how that affects it. I also had a additional question on how the rate translation block works. Say for example my agent has a sample time of 0.1 and my fixed solver has a step size of 0.01. Say my agent takes in the state from the environment then outputs the action. Since my solver step size is 10 times smaller than the agent sample time., is it the case my solver will update 10 times and then on the 10th update pass the resulting state as the next state into my agent?

[Torsten]

Since my solver step size is 10 times smaller than the agent sample time., is it the case my solver will update 10 times and then on the 10th update pass the resulting state as the next state into my agent?

I have no experience with Simulink. Maybe you should ask this as a new question in the forum and/or ask MATLAB Support.

Answer 1

[Chuguang Pan]

You can use Simulink Debug Functionality to step through the simulation and examine the output of agent.