Can I read Learning to Mix n-Step Returns: Generalizing lambda-Returns for Deep Reinforcement Learning on EtoBox?

Learning to Mix n-Step Returns: Generalizing lambda-Returns for Deep Reinforcement Learning by Sharma, Sahil; J, Girish Raguvir; Ramesh, Srivatsan; Ravindran, Balaraman is a scholarly article available to read on EtoBox.

What is Learning to Mix n-Step Returns: Generalizing lambda-Returns for Deep Reinforcement Learning about?

Reinforcement Learning (RL) can model complex behavior policies for goal-directed sequential decision making tasks. A hallmark of RL algorithms is Temporal Difference (TD) learning: value function for the current state is moved towards a bootstrapped target that is estimated using next state's value function. $\lambda$-returns generalize beyond 1-step returns and strike a balance between Monte Carlo and TD learning methods. While lambda-returns have been extensively studied in RL, they haven't been explored a lot in Deep RL. This paper's first contribution is an exhaustive benchmarking of lambda-returns. Although mathematically tractable, the use of exponentially decaying weighting of n-step returns based targets in lambda-returns is a rather ad-hoc design choice. Our second major contribution is that we propose a generalization of lambda-returns called Confidence-based Autodidactic Returns (CAR), wherein the RL agent learns the weighting of the n-step returns in an end-to-end manner. This allows the agent to learn to decide how much it wants to weigh the n-step returns based targets. In contrast, lambda-returns restrict RL agents to use an exponentially decaying weighting scheme.

Author

Sharma, Sahil; J, Girish Raguvir; Ramesh, Srivatsan; Ravindran, Balaraman

Published

2017

Language

EN