Can I read Learning Rate Transfer in µP Networks on EtoBox?

Learning Rate Transfer in µP Networks by nikebeta is a document available to read on EtoBox.

What is Learning Rate Transfer in µP Networks about?

This document presents the first proof of learning rate transfer in linear multi-layer perceptrons (MLPs) using the Maximal Update Parametrization (µP), demonstrating that the optimal learning rate converges to a non-zero constant as the network width increases. In contrast, alternative parametrizations like Standard Parametrization (SP) do not exhibit this property, leading to a shift in optimal learning rates towards zero. The findings are supported by both theoretical proofs and extensive empirical resul

Author

nikebeta

Language

EN