Suddenly the reward value goes wrong
2 views (last 30 days)
We are conducting reinforcement learning for two joint controllers.
Reinforcement learning method is soft actor critic.
The error between the result due to the action of the Agent and the target value is set as the reward value. In other words, the maximum reward value you can get is 0.
At the beginning of learning, it seems to be working well, but at the end of learning, an abnormal reward value is suddenly output.
What's the reason? In the simulation where the joint controller throws a ball, the reward function is the error between the flight distance of the ball and the target point below.
As a reward, an impossible value is output, is this a learning problem? Or is it another issue?
function s = Ayameperformance_func(Angle1, Angle2, V2)
X1 = sin(Angle1); % X coordinate of joint 1
Y1 = -cos(Angle1); % Y coordinate of joint 1
X2 = X1 + sin(Angle1+Angle2); % X coordinate of joint 2
Y2 = Y1 - cos(Angle1+Angle2); % Y coordinate of joint 2
Ball_V = abs(V2 * cos(Angle2)); % ball's initial velocity
Ball_Time = sqrt(2*abs(Y2)/9.8); % time for ball flight
s = X2 + abs(Ball_V)*Ball_Time; % ball distance
%% agent options
agentOptions = rlSACAgentOptions;
agentOptions.SampleTime = Ts;
agentOptions.DiscountFactor = 0.90;
agentOptions.TargetSmoothFactor = 1e-3;
agentOptions.ExperienceBufferLength = 1e6;
agentOptions.MiniBatchSize = 64;
agentOptions.EntropyWeightOptions.TargetEntropy = -3;
agentOptions.ResetExperienceBufferBeforeTraining = false;
agentOptions.TargetUpdateFrequency = 1;
agentOptions.NumWarmStartSteps = 64;
agent = rlSACAgent(actor,[critic1 critic2],agentOptions);
maxepisodes = 15000;
maxsteps = 1e6;
trainingOptions = rlTrainingOptions(...
trainingOptions.UseParallel = true;
trainingOptions.ParallelizationOptions.Mode = 'sync';
trainingOptions.ParallelizationOptions.StepsUntilDataIsSent = 50;
trainingOptions.ParallelizationOptions.DataToSendFromWorkers = 'Experiences';