About this document
The Structural Origin of Attention Sink: Variance Discrepancy, Super Neurons, and Dimension Disparity by cursonivel001 is a document available to read on EtoBox.
This document investigates the structural origins of the attention sink phenomenon in Large Language Models (LLMs), identifying it as a result of variance discrepancies in the self-attention mechanism. The authors propose that the initial token
- Author
- cursonivel001
- Language
- EN