Theoretical Basis and References for Hybrid SAC Example

Dear MathWorks Team,

I am currently working on a research project in which I plan to use the Hybrid Soft Actor-Critic (SAC) implementation shown in this example:

https://www.mathworks.com/help/reinforcement-learning/ug/train-hybrid-sac-for-path-following-control.html

I would like to ask for clarification on the theoretical background of this hybrid SAC method. Specifically:

1. Is there any academic reference or official documentation that explains the algorithmic basis of this hybrid version?
2. Has the standard SAC algorithm been modified to handle hybrid (discrete + continuous) action spaces in this implementation? If so, could you indicate what methodology was followed or which reference supports it?

I need to accurately cite the method in a journal publication, so any relevant details or references would be helpful.

Best regards

Answer 1

[William Rose]

@keyvan,

Here are some citaitons from Matlab Help and elsewhere. For arXiv publications, you cna search for publicaiotns in peer-revieweed journals also. Christodoulou (2019) generalizes the Soft Actor Critic algorithm to include discrete as well as continuous actions. Zhou (2022) is nother that you will probably find useful.

• Haarnoja, Tuomas, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, et al. “Soft Actor-Critic Algorithms and Applications.” arXiv, January 29, 2019. https://arxiv.org/abs/1812.05905.
• Haarnoja, Tuomas, Aurick Zhou, et al. "Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor." Proceedings of the 35 th International Conference on Machine Learning, Stockholm, Sweden, PMLR 80, 2018. https://proceedings.mlr.press/v80/haarnoja18b/haarnoja18b.pdf
• Christodoulou, Petros. “Soft Actor-Critic for Discrete Action Settings.” arXiv, October 18, 2019. https://arxiv.org/abs/1910.07207.
• Zhou et al. "Revisiting Discrete Soft Actor-Critic" arXiv, 21 Sep 2022. https://arxiv.org/abs/2209.10081
• Delalleau, Olivier, Maxim Peter, Eloi Alonso, and Adrien Logut. “Discrete and Continuous Action Representation for Practical RL in Video Games.” arXiv, December 23, 2019. https://arxiv.org/abs/1912.11077

[keyvan]

Thank you kindly

[William Rose]

@keyvan, you are welcome. Good luck with your research.