Can I read RAIN: Your Language Models Can Align Themselves without Finetuning on EtoBox?

RAIN: Your Language Models Can Align Themselves without Finetuning by Li, Yuhui; Wei, Fangyun; Zhao, Jinjing; Zhang, Chao; Zhang, Hongyang is a scholarly article available to read on EtoBox.

What is RAIN: Your Language Models Can Align Themselves without Finetuning about?

Large language models (LLMs) often demonstrate inconsistencies with human preferences. Previous research typically gathered human preference data and then aligned the pre-trained models using reinforcement learning or instruction tuning, a.k.a. the finetuning step. In contrast, aligning frozen LLMs without requiring alignment data is more appealing. This work explores the potential of the latter setting. We discover that by integrating self-evaluation and rewind mechanisms, unaligned LLMs can directly produce responses consistent with human preferences via self-boosting. We introduce a novel inference method, Rewindable Auto-regressive INference (RAIN), that allows pre-trained LLMs to evaluate their own generation and use the evaluation results to guide rewind and generation for AI safety. Notably, RAIN operates without the need of extra data for model alignment and abstains from any training, gradient computation, or parameter updates. Experimental results evaluated by GPT-4 and humans demonstrate the effectiveness of RAIN: on the HH dataset, RAIN improves the harmlessness rate of LLaMA 30B from 82% of vanilla inference to 97%, while maintaining the helpfulness rate. On the Truthf

Author

Li, Yuhui; Wei, Fangyun; Zhao, Jinjing; Zhang, Chao; Zhang, Hongyang

Published

2023

Language

EN