Deep Reinforcement Learning Toolbox with LSTM RNN

Hi,

Currently I am using a LSTM RNN network in my TD3 agent model.

I am using a LSTM network architecture based on the DDPG TD3 the online help, as shown below

%Critic Network

rng(0)  % fix the random seed

statePath1 = [
    sequenceInputLayer(numObservations,'Normalization','none','Name','observation')
    fullyConnectedLayer(400,'Name','CriticStateFC1')
    reluLayer('Name','CriticStateRelu1')
    fullyConnectedLayer(300,'Name','CriticStateFC2')
    ];
actionPath1 = [
    sequenceInputLayer(numActions,'Normalization','none','Name','action')
    fullyConnectedLayer(300,'Name','CriticActionFC1')
    ];
commonPath1 = [
    additionLayer(2,'Name','add')
    reluLayer('Name','CriticCommonRelu1')
    lstmLayer(16,'OutputMode','sequence','Name','CriticLSTM');
    fullyConnectedLayer(1,'Name','CriticOutput')
    ];

criticNet = layerGraph(statePath1);
criticNet = addLayers(criticNet,actionPath1);
criticNet = addLayers(criticNet,commonPath1);
criticNet = connectLayers(criticNet,'CriticStateFC2','add/in1');
criticNet = connectLayers(criticNet,'CriticActionFC1','add/in2');

%Actor Network
actorNet = [
    sequenceInputLayer(numObservations,'Normalization','none','Name','observation')
    fullyConnectedLayer(400,'Name','ActorFC1')
    lstmLayer(8,'OutputMode','sequence','Name','ActorLSTM')
    reluLayer('Name','ActorRelu1')
    fullyConnectedLayer(300,'Name','ActorFC2')
    reluLayer('Name','ActorRelu2')
    fullyConnectedLayer(numActions,'Name','ActorFC3')                       
    tanhLayer('Name','ActorTanh1')
    ];

My Mini Batch size is set to 200.

When my training begins the model runs very quickly, during the initial episodes, which is the same as the mini batch size of 200, then from episode 201 onward the model runs at a normal training pace and seems to start learning, this can be seen in the episode manager plot below.

My question is:

Is this initial training behavour normal for a LSTM RNN?

Cheers

Patrick