Can I read Rethinking Population-assisted Off-policy Reinforcement Learning on EtoBox?

Rethinking Population-assisted Off-policy Reinforcement Learning by Zheng, Bowen; Cheng, Ran is a scholarly article available to read on EtoBox.

What is Rethinking Population-assisted Off-policy Reinforcement Learning about?

While off-policy reinforcement learning (RL) algorithms are sample efficient due to gradient-based updates and data reuse in the replay buffer, they struggle with convergence to local optima due to limited exploration. On the other hand, population-based algorithms offer a natural exploration strategy, but their heuristic black-box operators are inefficient. Recent algorithms have integrated these two methods, connecting them through a shared replay buffer. However, the effect of using diverse data from population optimization iterations on off-policy RL algorithms has not been thoroughly investigated. In this paper, we first analyze the use of off-policy RL algorithms in combination with population-based algorithms, showing that the use of population data could introduce an overlooked error and harm performance. To test this, we propose a uniform and scalable training design and conduct experiments on our tailored framework in robot locomotion tasks from the OpenAI gym. Our results substantiate that using population data in off-policy RL can cause instability during training and even degrade performance. To remedy this issue, we further propose a double replay buffer design that p

Author

Zheng, Bowen; Cheng, Ran

Published

2023

Language

EN