Can I read Mixture-of-Recursions for Efficient Computation on EtoBox?
Mixture-of-Recursions for Efficient Computation by violist32 is a document available to read on EtoBox.
What is Mixture-of-Recursions for Efficient Computation about?
The document introduces Mixture-of-Recursions (MoR), a framework that combines parameter sharing and adaptive computation in Recursive Transformers to enhance efficiency in language modeling. MoR employs lightweight routers to dynamically assign recursion depths to individual tokens, optimizing memory access and reducing computational demands. Empirical results show that MoR significantly improves validation perplexity and few-shot accuracy across various model sizes while maintaining lower training costs.
- Author
- violist32
- Language
- EN