Can I read Training Language Models with Language Feedback at Scale on EtoBox?

Training Language Models with Language Feedback at Scale by Scheurer, Jérémy; Campos, Jon Ander; Korbak, Tomasz; Chan, Jun Shern; Chen, Angelica; Cho, Kyunghyun; Perez, Ethan is a scholarly article available to read on EtoBox.

What is Training Language Models with Language Feedback at Scale about?

Pretrained language models often generate outputs that are not in line with human preferences, such as harmful text or factually incorrect summaries. Recent work approaches the above issues by learning from a simple form of human feedback: comparisons between pairs of model-generated outputs. However, comparison feedback only conveys limited information about human preferences. In this paper, we introduce Imitation learning from Language Feedback (ILF), a new approach that utilizes more informative language feedback. ILF consists of three steps that are applied iteratively: first, conditioning the language model on the input, an initial LM output, and feedback to generate refinements. Second, selecting the refinement incorporating the most feedback. Third, finetuning the language model to maximize the likelihood of the chosen refinement given the input. We show theoretically that ILF can be viewed as Bayesian Inference, similar to Reinforcement Learning from human feedback. We evaluate ILF's effectiveness on a carefully-controlled toy task and a realistic summarization task. Our experiments demonstrate that large language models accurately incorporate feedback and that finetuning w

Author

Scheurer, Jérémy; Campos, Jon Ander; Korbak, Tomasz; Chan, Jun Shern; Chen, Angelica; Cho, Kyunghyun; Perez, Ethan

Published

2023

Language

EN