Can I read A Note on the Convergence of Muon and Further on EtoBox?

A Note on the Convergence of Muon and Further by Li, Jiaxiang; Hong, Mingyi is a scholarly article available to read on EtoBox.

What is A Note on the Convergence of Muon and Further about?

In this note, we inspect the convergence of a new optimizer for pretraining LLMs, namely the Muon optimizer. Such an optimizer is closely related to a specialized steepest descent method where the update direction is the minimizer of the quadratic approximation of the objective function under spectral norm. We provide the convergence analysis on both versions of the optimizer and discuss its implications.

Author

Li, Jiaxiang; Hong, Mingyi

Published

2025

Language

EN