rlSARSAAgent

R2026b

SARSA reinforcement learning agent

Description

The SARSA algorithm is an on-policy reinforcement learning method for environments with a discrete action space. A SARSA agent trains a Q-value function critic to estimate the value of the current epsilon-greedy policy (it does not try to directly learn an optimal policy). SARSA agents do not support recurrent neural networks.

For more information on SARSA agents, see SARSA Agent.

For more information on the different types of reinforcement learning agents, see Reinforcement Learning Agents.

Creation

Syntax

agent = rlSARSAAgent(observationInfo,actionInfo)

agent = rlSARSAAgent(observationInfo,actionInfo,initOpts)

agent = rlSARSAAgent(critic,agentOptions)

Description

Create Default Agent from Observation and Action Specifications

agent = rlSARSAAgent(observationInfo,actionInfo) creates a SARSA agent for an environment with the given observation and action specifications, using default initialization options. The critic in the agent uses a table (if the observation has only one, discrete, channel) or deep neural network (otherwise).

agent = rlSARSAAgent(observationInfo,actionInfo,initOpts) creates a SARSA agent for an environment with the given observation and action specifications. When the agent uses a default network, each hidden fully connected layer has the number of units specified in the initOpts object. When the agent uses a table, initOpts is ignored. SARSA agents do not support recurrent networks. For more information on the initialization options, see rlAgentInitializationOptions.

Create Agent from Critic

agent = rlSARSAAgent(critic,agentOptions) creates a SARSA agent with the specified critic network and sets the AgentOptions property.

Input Arguments

observationInfo — Observation specifications

rlFiniteSetSpec object | rlNumericSpec object | array

Observation specifications, specified as an rlFiniteSetSpec or rlNumericSpec object or an array containing any combination of such objects. Each element in the array defines the properties of an environment observation channel, such as its dimensions, data type, and name.

This argument sets the ObservationInfo property.

Example: observationInfo=[rlNumericSpec([2 1]) rlFiniteSetSpec([-1 1])]

actionInfo — Action specification

rlFiniteSetSpec object

Action specification, specified as an rlFiniteSetSpec object. This object defines the properties of the environment action channel, such as its dimensions, data type, and name.

This argument sets the ActionInfo property.

Example: actionInfo=rlFiniteSetSpec([-1 0 1])

initOpts — Agent initialization options

rlAgentInitializationOptions object

Agent initialization options, specified as an rlAgentInitializationOptions object.

Example: rlAgentInitializationOptions(NumHiddenUnit=128)

critic — Critic

rlQValueFunction object

Critic, specified as an rlQValueFunction object. For more information on creating critics, see Create Actors, Critics, and Policy Objects.

agentOptions — Agent options

rlSARSAAgentOptions object

Agent options, specified as an rlSARSAAgentOptions object.

This argument sets the AgentOptions property.

Example: rlSARSAAgentOptions(DiscountFactor=0.9)

Properties

AgentOptions — Agent options

rlSARSAAgentOptions object

Agent options, specified as an rlSARSAAgentOptions object.

Example: myagent.AgentOptions = rlSARSAAgentOptions(DiscountFactor=0.9)

UseExplorationPolicy — Option to use exploration policy for simulation and deployment

false (default) | true

Option to use an exploration policy when selecting actions during simulation or after deployment, specified as a logical value.

• true — Specify this value to use the base agent exploration policy when you use the agent with the sim and generatePolicyFunction functions. Specifically, in this case, the agent uses the rlEpsilonGreedyPolicy object. The action selection has a random component, so the agent explores its action and observation spaces.

• false — Specify this value to force the agent to use the base agent greedy policy (the action with maximum likelihood) when you use the agent with the sim and generatePolicyFunction functions. Specifically, in this case, the agent uses the rlMaxQPolicy policy. The action selection is greedy, so the policy behaves deterministically and the agent does not explore its action and observation spaces.

Note

This option affects only simulation and deployment and does not affect training. When you train an agent using the train function, the agent always uses its exploration policy independently of the value of this property. Specifically, the training algorithm temporarily sets UseExplorationPolicy to true for the duration of the training,and then reverts it to the original value when the training is completed.

Example: myagent.UseExplorationPolicy = true

ObservationInfo — Observation specifications

Read-only: specification object | array of specification objects

This property is read-only.

Observation specifications, returned as an rlFiniteSetSpec or rlNumericSpec object or an array containing any combination of such objects. Each element in the array defines the properties of an environment observation channel, such as its dimensions, data type, and name.

If you create the agent by specifying an actor or critic, the value of ObservationInfo matches the value specified in the actor and critic objects. If you create a default agent, the agent constructor function sets the ObservationInfo property to the input argument observationInfo.

You can extract observationInfo from an existing environment, function approximator, or agent using getObservationInfo. You can also construct the specifications manually using rlFiniteSetSpec or rlNumericSpec.

ActionInfo — Action specification

Read-only: rlFiniteSetSpec object

This property is read-only.

Action specifications, specified as an rlFiniteSetSpec object. This object defines the properties of the environment action channel, such as its dimensions, data type, and name.

Note

For this agent, only one action channel is allowed.

If you create the agent by specifying a critic object, the value of ActionInfo matches the value specified in critic. If you create a default agent, the agent constructor function sets the ActionInfo property to the input argument ActionInfo.

You can extract actionInfo from an existing environment, function approximator, or agent using getActionInfo. You can also construct the specification manually using rlFiniteSetSpec.

SampleTime — Sample time of agent

1 (default) | positive scalar | -1

Sample time of the agent, specified as a positive scalar or as -1.

Within a MATLAB^® environment, the agent is executed every time the environment advances, so, SampleTime does not affect the timing of the agent execution. If SampleTime is set to -1, in MATLAB environments, the time interval between consecutive elements in the returned output experience is considered equal to 1.

Within a Simulink^® environment, the RL Agent block that uses the agent object executes every SampleTime seconds of simulation time. If SampleTime is set to -1 the block inherits the sample time from its input signals. Set SampleTime to -1 when the block is a child of an event-driven subsystem.

Set SampleTime to a positive scalar when the block is not a child of an event-driven subsystem. Doing so ensures that the block executes at appropriate intervals when input signal sample times change due to model variations. If SampleTime is a positive scalar, this value is also the time interval between consecutive elements in the output experience returned by sim or train, regardless of the type of environment.

If SampleTime is set to -1, in Simulink environments, the time interval between consecutive elements in the returned output experience reflects the timing of the events that trigger the RL Agent block execution.

This property is shared between the agent and the agent options object within the agent. If you change this property in the agent options object, it also changes in the agent, and vice versa.

Example: myagent.SampleTime = -1 sets the sample time of the agent object myagent to -1.

Object Functions

train	Train reinforcement learning agents within a specified environment
sim	Simulate trained reinforcement learning agents within specified environment
getAction	Obtain action from agent, actor, or policy object given environment observations
getCritic	Extract critic from reinforcement learning agent
setCritic	Set critic of reinforcement learning agent
generatePolicyFunction	Generate MATLAB function that evaluates policy of an agent or policy object

Examples

Create Default SARSA Agent

Create or load an environment object with a discrete action space. For this example, use the same environment as in the example Train Reinforcement Learning Agent in Basic Grid World.

env = rlPredefinedEnv("BasicGridWorld");

Get the observation and action specifications.

obsInfo = getObservationInfo(env);
actInfo = getActionInfo(env);

Create a Q-learning agent using the specifications.

agent = rlSARSAAgent(obsInfo,actInfo)

agent = 
  rlSARSAAgent with properties:

            AgentOptions: [1×1 rl.option.rlSARSAAgentOptions]
    UseExplorationPolicy: 0
         ObservationInfo: [1×1 rl.util.rlFiniteSetSpec]
              ActionInfo: [1×1 rl.util.rlFiniteSetSpec]
              SampleTime: 1

Because the observation is a single discrete channel, the agent's critic uses a table as approximation model.

getModel(getCritic(agent))

ans = 
  rlTable with properties:

    Table: [25×4 double]

Specify an Epsilon value of 0.2.

agent.AgentOptions.EpsilonGreedyExploration.Epsilon = 0.2;

To check your agent, use the getAction function to return the action from a batch of 10 random observations.

act = getAction(agent,{randi(numel(obsInfo.Elements),[obsInfo.Dimension(1) 10])});

Display the seventh element in the batch.

a = act{1}(7)

a = 
1

To find the action label corresponding to the numerical index, use the idx2action function.

idx2action(env.Model,a)

ans = 
"N"

The action represents an attempt to move north.

You can now test and train the agent against the environment.

Create Default SARSA Agent Using Initialization Options

Create an environment with a discrete action space, and obtain its observation and action specifications.

For this example, load the environment used in the example Create DQN Agent Using Deep Network Designer and Train Using Image Observations. This environment has two observation channels: one carrying a 50-by-50 grayscale image and the other carrying a scalar (the angular velocity of the pendulum). The action is a scalar with five possible elements (a torque of either -2, -1, 0, 1, or 2 Nm applied to a swinging pole).

% Load predefined environment.
env = rlPredefinedEnv("SimplePendulumWithImage-Discrete");

% Obtain observation and action specifications.
obsInfo = getObservationInfo(env);
actInfo = getActionInfo(env);

Create an agent initialization option object, specifying that each hidden fully connected layer in the network must have 128 neurons (instead of the default number, 256).

initOpts = rlAgentInitializationOptions(NumHiddenUnit=128);

The agent creation function initializes the actor and critic networks randomly. To reproduce the results of this section, specify the seed and algorithm used for random number generation.

rng(0,"twister")

Create a default SARSA agent from the environment observation and action specifications. Pass the initialization options object as the third argument.

agent = rlSARSAAgent(obsInfo,actInfo,initOpts);

Extract the deep neural network from both the critic.

criticNet = getModel(getCritic(agent));

To verify that each hidden fully connected layer has 128 neurons, you can display the layers on the MATLAB® command window,

criticNet.Layers

or visualize the structure interactively using analyzeNetwork.

analyzeNetwork(criticNet)

Plot the critic network.

plot(criticNet)

To check your agent, use the getAction function to return the actions from a batch of 10 random observations.

obs1 = rand([obsInfo(1).Dimension 10]);
obs2 = rand([obsInfo(2).Dimension 10]);
act = getAction(agent,{obs1,obs2});

Display the seventh element in the batch.

act{1}(7)

ans = 
-1

You can now test and train the agent within the environment.

Create SARSA Agent from Critic

Create or load an environment object with a discrete action space. For this example load the Basic Grid World environment object also used in the example Train Reinforcement Learning Agent in Basic Grid World.

env = rlPredefinedEnv("BasicGridWorld");

Get the observation and action specifications.

obsInfo = getObservationInfo(env);
actInfo = getActionInfo(env);

A SARSA agent uses a parameterized Q-value function to estimate the value of the policy. A Q-value function takes the current observation and an action as inputs and returns a single scalar as output (the estimated discounted cumulative long-term reward for taking the action from the state corresponding to the current observation, and following the policy thereafter).

Because in this example both the observation and action spaces are discrete and low-dimensional, use a table to model the Q-value function within the critic. rlTable creates a value table object from the observation and action specifications objects.

Create a table approximation model derived from the environment observation and action specifications.

qTable = rlTable(obsInfo,actInfo);

Create the Q-value function approximator object using qTable and the environment specification objects. For more information, see rlQValueFunction.

critic = rlQValueFunction(qTable,obsInfo,actInfo);

Create a SARSA agent using the approximator object.

agent = rlSARSAAgent(critic)

agent = 
  rlSARSAAgent with properties:

            AgentOptions: [1×1 rl.option.rlSARSAAgentOptions]
    UseExplorationPolicy: 0
         ObservationInfo: [1×1 rl.util.rlFiniteSetSpec]
              ActionInfo: [1×1 rl.util.rlFiniteSetSpec]
              SampleTime: 1

Specify an Epsilon value of 0.2.

agent.AgentOptions.EpsilonGreedyExploration.Epsilon = 0.2;

To check your agent, use the getAction function to return the action from a batch of 10 random observations.

act = getAction(agent,{randi(numel(obsInfo.Elements),[obsInfo.Dimension(1) 10])});

Display the seventh element in the batch.

a = act{1}(7)

a = 
1

To find the action label corresponding to the numerical index, use the idx2action function.

idx2action(env.Model,a)

ans = 
"N"

The action represents an attempt to move north.

You can now test and train the agent against the environment.

Version History

Introduced in R2019a