Can I read Coherent Wave Dynamics and Language Generation of a Generative Pre-trained Transformer on EtoBox?

Coherent Wave Dynamics and Language Generation of a Generative Pre-trained Transformer by Hong, Tao is a scholarly article available to read on EtoBox.

What is Coherent Wave Dynamics and Language Generation of a Generative Pre-trained Transformer about?

Large Language Models (LLMs), such as the Generative Pretrained Transformer (GPT), have achieved tremendous success in various language tasks, but their emergent abilities have also raised many questions, concerns, and challenges that need to be addressed. To gain a better understanding of the models' inner mechanisms, we analyze the hidden state and channel wave dynamics in a small GPT, focusing on the coherence of wave patterns in terms of cross-channel correlation and individual auto-correlation. Our findings suggest that wave dynamics offer consistent and repeatable intrinsic oscillation modes, along with context-aware plasticity and expressiveness in language generation. By analyzing wave patterns, coherence, and clustering, we provide a systematic way to identify and interpret the functionality of the hidden state channels, paving the way to understand and control higher-level language pattern formation. In addition, we investigate the Poisson statistics of spelling errors in text sequence generation across various levels of model training and observe a phase-transition-like process. As coherence builds up, there is a competition between the generation of correct and misspell

Author

Hong, Tao

Published

2023

Language

EN