Can I read Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMs on EtoBox?

Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMs by Guo, Tianyu; Pai, Druv; Bai, Yu; Jiao, Jiantao; Jordan, Michael I.; Mei, Song is a scholarly article available to read on EtoBox.

What is Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMs about?

Practitioners have consistently observed three puzzling phenomena in transformer-based large language models (LLMs): attention sinks, value-state drains, and residual-state peaks, collectively referred to as extreme-token phenomena. These phenomena are characterized by certain so-called "sink tokens" receiving disproportionately high attention weights, exhibiting significantly smaller value states, and having much larger residual-state norms than those of other tokens. These extreme tokens give rise to various challenges in LLM inference, quantization, and interpretability. We elucidate the mechanisms behind extreme-token phenomena. First, we show that these phenomena arise in very simple architectures -- transformers with one to three layers -- trained on a toy model, the Bigram-Backcopy (BB) task. In this setting, we identify an active-dormant mechanism, where attention heads become sinks for specific input domains while remaining non-sinks for others. Our theoretical analysis of the training dynamics reveals that these phenomena are driven by a mutual reinforcement mechanism. Building on these insights, we propose strategies to mitigate extreme-token phenomena during pretraining

Author

Guo, Tianyu; Pai, Druv; Bai, Yu; Jiao, Jiantao; Jordan, Michael I.; Mei, Song

Published

2024

Language

EN