TD3 error message for using ltsm layer in Neural Network

Hi! I am trying to design a reinforcement learning model for landing mission on the moon in a defined region. I played with different Agents algorithm such as PPO, DDPG and TD3 to evaluate how they work differently.

With PPO I don't have problems related to error in the code or in the network architecture so at the moment I am working with it. The problem is when I try to use DDPG and TD3 with recurrent neural network, including an lstm layer in the architecture, I obtain the following error message:

Error using dlnetwork/predict (line 664)
Layer 'lstm': Invalid input data. Input data must contain a dimension labeled 'T' and must not contain any non-singleton dimensions labeled 'U'.

Error in rl.representation.model.rlDLNetworkModel/cacheNetworkSize (line 588)
            [DummyOutput{:}] = predict(this.InternalNetwork,DummyInput{:},'Acceleration','none');

Error in rl.representation.model.rlDLNetworkModel (line 90)
            this = cacheNetworkSize(this);

Error in rl.util.createInternalModelFactory (line 16)
                Model = rl.representation.model.rlDLNetworkModel(Model, UseDevice, ObservationNames, ActionNames);

Error in rlDeterministicActorRepresentation (line 86)
Model = rl.util.createInternalModelFactory(Model, Options, ObservationNames, ActionNames, InputSize, OutputSize);

Error in agentCreator (line 236)
        actor = rlDeterministicActorRepresentation(actnet,obsInfo,actInfo,"Observation","obs","Action","fcact",opts)

Error in main (line 26)
[agent] = agentCreator(numObs,obsInfo,obsInfocr,numAct,actInfo,'TD3_recurrent');
 

The Neural Network I am trying to implement is the following:

        actnet = [featureInputLayer(numObs,"Name","obs"); 
            fullyConnectedLayer(50,"Name","fc1");
            fullyConnectedLayer(30,"Name","fc2");
            reluLayer('Name','relu1');
            lstmLayer(8,'OutputMode','sequence','Name','lstm')
            fullyConnectedLayer(4,"Name","fcact")];

I looked for documentation but I cannot find any help.

Can someone clarify this for me?

Thaks!

[yanqi liu]

yes，sir，may be upload your data mat file，then we can debug it

[Francesco Mogetti]

The code is nothing more than this:

numObs = 6;

numAct = 4;

obsInfo = rlNumericSpec([numObs 1]);

actInfo = rlNumericSpec([numAct 1]);

actInfo.UpperLimit = 1;

actInfo.LowerLimit = -1;

mdl = "EnvTD3";

env = rlSimulinkEnv(mdl,"EnvTD3/RL Agent",obsInfo,actInfo);

% env.ResetFcn = @ResetLanderCond2;

[agent] = agentCreator(numObs,obsInfo,numAct,actInfo,'TD3');

Then in the function "agentCreator" the neural network is defined both for actor and critic. The problem is related to the actor which is defined first:

actnet = [featureInputLayer(numObs,"Name","obs");

fullyConnectedLayer(50,"Name","fc1");

fullyConnectedLayer(30,"Name","fc2");

reluLayer('Name','relu1');

lstmLayer(8,'OutputMode','sequence','Name','lstm')

fullyConnectedLayer(4,"Name","fcact")];

opts = rlRepresentationOptions("Learnrate",5e-4,"GradientThreshold",10);

actor = rlDeterministicActorRepresentation(actnet,obsInfo,actInfo,"Observation","obs","Action","fcact",opts)

So the error in the command window:

Error using dlnetwork/predict (line 664)
Layer 'lstm': Invalid input data. Input data must contain a dimension labeled 'T' and must not contain any non-singleton dimensions labeled 'U'.

Error in rl.representation.model.rlDLNetworkModel/cacheNetworkSize (line 588)
            [DummyOutput{:}] = predict(this.InternalNetwork,DummyInput{:},'Acceleration','none');

Error in rl.representation.model.rlDLNetworkModel (line 90)
            this = cacheNetworkSize(this);

Error in rl.util.createInternalModelFactory (line 16)
                Model = rl.representation.model.rlDLNetworkModel(Model, UseDevice, ObservationNames, ActionNames);

Error in rlDeterministicActorRepresentation (line 86)
Model = rl.util.createInternalModelFactory(Model, Options, ObservationNames, ActionNames, InputSize, OutputSize);

Error in agentCreator (line 236)
        actor = rlDeterministicActorRepresentation(actnet,obsInfo,actInfo,"Observation","obs","Action","fcact",opts)

Error in main (line 26)
[agent] = agentCreator(numObs,obsInfo,obsInfocr,numAct,actInfo,'TD3_recurrent');
 

Answer 1

[Pratyush Roy]

Hi Francesco,

The input format for the data which is passed to an lstmLayer can be either CBT (channel, batch, time), SCBT(spatial, channel, batch, time) or SSCBT(spatial,spatial, channel, batch, time) or SSSCBT(spatial, spatial, spatial, channel, batch, time) (please refer to this documentation).

So we should make sure that the data obtained from the previous layers is changed to the aforementioned formats before feeding that to the lstmLayer(please refer to the documentation here to more information on data formats).

Another workaround would be to change the architecture of the actor network by introducing a sequenceInputLayer followed by the lstmLayer, which might be followed by more layers.

Currently we are aware of this issue which shows the error message and our developers might address this in one of our future releases.

Hope this helps!