Can I read SEAL: Systematic Error Analysis for Value ALignment on EtoBox?

SEAL: Systematic Error Analysis for Value ALignment by Revel, Manon; Cargnelutti, Matteo; Eloundou, Tyna; Leppert, Greg is a scholarly article available to read on EtoBox.

What is SEAL: Systematic Error Analysis for Value ALignment about?

Reinforcement Learning from Human Feedback (RLHF) aims to align language models (LMs) with human values by training reward models (RMs) on binary preferences and using these RMs to fine-tune the base LMs. Despite its importance, the internal mechanisms of RLHF remain poorly understood. This paper introduces new metrics to evaluate the effectiveness of modeling and aligning human values, namely feature imprint, alignment resistance and alignment robustness. We categorize alignment datasets into target features (desired values) and spoiler features (undesired concepts). By regressing RM scores against these features, we quantify the extent to which RMs reward them - a metric we term feature imprint. We define alignment resistance as the proportion of the preference dataset where RMs fail to match human preferences, and we assess alignment robustness by analyzing RM responses to perturbed inputs. Our experiments, utilizing open-source components like the Anthropic/hh-rlhf preference dataset and OpenAssistant RMs, reveal significant imprints of target features and a notable sensitivity to spoiler features. We observed a 26% incidence of alignment resistance in portions of the dataset w

Author

Revel, Manon; Cargnelutti, Matteo; Eloundou, Tyna; Leppert, Greg

Published

2024

Language

EN