Can I read SOAP: Efficient Shampoo Optimization on EtoBox?
SOAP: Efficient Shampoo Optimization by yunitaismawati046 is a document available to read on EtoBox.
What is SOAP: Efficient Shampoo Optimization about?
This document presents SOAP (ShampoO with Adam in the Preconditioner’s eigenbasis), a new optimization algorithm that improves upon Shampoo by reducing computational overhead and hyperparameters while maintaining performance. SOAP demonstrates significant efficiency gains in language model pre-training tasks, outperforming both Shampoo and Adam, with reductions in iterations and wall clock time. The authors establish a formal connection between Shampoo and Adafactor, leading to the development of SOAP, whic
- Author
- yunitaismawati046
- Language
- EN