Multi-Step (D)DQN using Parallelization

I have noticed that with the current implementation of DQN it is not possible to use both multi-step returns (NumStepsToLookAhead>1) and parallelization. However I noticed that having multi-steps is essential for my application. Still, I would love to make use of all of my cpu cores.

Thus, I am wondering if it is possible to implement a custom DQN agent that allows for this. My goal is to arrive at an implementation where multiple workers generate experience samples. Learning is performed centrally and the updated policy is returned to the worker regularily.

Is this a reasonable idea? If yes, does anybody have an idea how I can implement this without generating too much code duplicates of the default dqn implementation?

Thank you very much.

Answer 1

[Ayush Modi]

Hi David,

As per my understanding, you would like to generate experience samples at worker nodes by training the model with local data and then send these model parameters to central server to train the central model.

You can achieve this by using the concept of Federated Learning.

Please refer to the following MathWorks documentation for more information on Federated Learning:

https://www.mathworks.com/help/deeplearning/ug/train-network-using-federated-learning.html

You can create a custom DQN/DDQN model as well.

I hope this resolves the issue you were facing.