Problems importing Farama Gymnasiums (previously Open AI gym) continuous environments in MATLAB to use RL toolbox

Dear all,

I am using RL toolbox in Matlab to solve Farama Gymnasium environments (https://gymnasium.farama.org/)

While in environments with Discrete action spaces, the procedure works well, in Environments with CONTINUOUS action spaces gives an error in the step function inside the needed wrapper class, in the second line.

The lines that throws the error are:

ActionSingle = single(Action);
result = this.open_env.step(ActionSingle);

The error I get is :

Error using rl.train.SeriesTrainer/run
There was an error executing the environment's step method.

Caused by:
Error using continuous_mountain_car>step
Python Error: TypeError: 'float' object is not subscriptable

I think this error has something to do with the integration of Python ndarrays in Matlab.

The same issue is reported here:

https://es.mathworks.com/matlabcentral/answers/568278-environment-for-reinforcement-learning-project?s_tid=srchtitle

How can I solve this problem so that I can train environments with continuous action spaces?

Thank you in advance,

Alberto

CODE FOR THE WRAPPER CLASS

classdef MountainCarContinuousEnv < rl.env.MATLABEnvironment
    %MYCARTPOLEENV: custom environment in MATLAB.    
    
    %% Properties (set properties' attributes accordingly)
    properties
        open_env = py.gymnasium.make("MountainCarContinuous-v0");     
    end
    %% Necessary Methods
    methods              
        function this = MountainCarContinuousEnv()
            ObservationInfo = rlNumericSpec([2 1],'LowerLimit',[-1.2 -0.07]', 'UpperLimit', [0.6 0.07]');
            ObservationInfo.Name = 'MountainCarObservation';
            ObservationInfo.Description = 'Position, Velocity';
            ActionInfo = rlNumericSpec([1 1],'LowerLimit',-1, 'UpperLimit', 1); %CONTINUOUS ACTION SPACE !!!!!
            %ActionInfo = setDataType(ActionInfo,'single');
            ActionInfo.Name = 'DirectionalForce';
            this = [e-mail](ObservationInfo, ActionInfo);
        end
        function [Observation,Reward,IsDone,LoggedSignals] = step(this,Action)
            ActionSingle = single(Action);
            result = this.open_env.step(ActionSingle); % PROBLEM HERE !!!!!
            Observation = double(result{1})'; 
            Reward = result{2};
            IsDone = result{3};
            LoggedSignals = [];
        end
        function InitialObservation = reset(this)
            result = this.open_env.reset();
            InitialObservation = double(result{1})';  
        end
        function result =render(this)
            result = this.open_env.render();
        end
        function result =close(this)
            result = this.open_env.close();
        end
    end
end

Answer 1

[Emmanouil Tzorakoleftherakis]

Hi Alberto,

In the post you are mentioning, I recommended a 3rd party tool to use OpenAI Gym with Reinforcement Learning Toolbox. For issues with this tool, I would normally recommend talking to the developer of the tool.

Nevertheless, in this case it seems to be a data conversion error. I think if you replace

result = this.open_env.step(ActionSingle);

with

result = this.open_env.step(py.numpy.array(ActionSingle,pyargs('ndmin',int32(1))));

the error will go away.

Also, as I mentioned in the other post, MathWorks is working on making available an interface from Reinforcement Learning Toolbox to OpenAI Gym, but I don't have any specific dates I can share yet.

Hope this helps