Specifying Actions using rlFiniteSetSpec or rlNumericSpec (Reinforcement Learning)

I understand that I can use CreateGridWorld for a Grid Maze environment for reinforcement learning. However, when creating my own, I have to specify the action settings using either rlFiniteSetSpec or rlNumericSpec. How am I to specify them if my actions are 4 (U, D, L, R). Any suggestions and help would be much appreciated!

Answer 1

[Rushi Vyas]

Hi Huzaifah,

It is my understanding that have 4 actions (U, D, L, R) and want to specify them in your RL agent.

Generally, rlNumericSpec is used when you have a continuous action space and rlFiniteSetSpec is used when you have a discrete action space.

In your case, the action space is discrete. So, you should go with rlFiniteSetSpec.

As rlFiniteSetSpec takes only numeric input, you have to encode your actions to numbers. For example, (U,D,L,R) -> (1,2,3,4)

>> actInfo = rlFiniteSetSpec([1,2,3,4])  
actInfo = 
  rlFiniteSetSpec with properties:
       Elements: [4×1 double]
           Name: [0×0 string]
    Description: [0×0 string]
      Dimension: [1 1]
       DataType: "double"

For more information you can refer to : https://in.mathworks.com/help/reinforcement-learning/ref/rl.util.rlfinitesetspec.html

[Huzaifah Shamim]

Ah ok that is what I realized. Thank You so much. My question with that then becomes: How/Where do I code that in my function that those numbers correspond to those 4 actions? I used the myEnvClass.m (the one given to us to edit) to define it but am unsure if that is correct.

Additionally, could I techincally use rlNumericSpec if I wanted two outputs, one for the XDirection and one for the YDirection? So my actInfo would be something like: rlNumericSpec([-5 5; -5 5]) where the first row indicates that I can go from -5 to 5 in the xdirection and the second row indicates that I can go from -5 to 5 in the ydirection. Is that allowed and would that work?

[Dongqi Liu]

Hi Huzaifah,

If you want to use rlFiniteSetSpec to specify 4 actions,you could try like this.

actInfo = rlFiniteSetSpec({[1 5],[1 6],[2 5],[2 6]})  

For example,this code can decide your first action in 1 or 2,and your second action in 5 or 6.

If there are 4 actions and 5 possbilities each action,you might create a 1x625 cell element to specify actinfo.

It is not a good way,but I hope thiscould help you.

[Dr. Ali Peker]

If we have a custom step function which will use these actions. How we can access individual actions inside actInfo for rlNumericSpec and rlFiniteSetSpec?

Answer 2

[jie lu]

open_system('rlwatertank')

Loading

...\matlab2021a\bin\win64\builtins\sl_main\mwlibmwsimulink_builtinimpl.dllfailed

with error: 找不到指定的程序。

: state not recoverable: state not recoverable

How to solve this problem?

Any suggestions and help would be much appreciated!