Can I read Super Consistency in Neural Network Landscapes on EtoBox?

Super Consistency in Neural Network Landscapes by Alexandru Meterez is a document available to read on EtoBox.

What is Super Consistency in Neural Network Landscapes about?

This paper introduces the concept of Super Consistency in neural network landscapes, demonstrating that certain spectral properties of the loss Hessian remain largely independent of network size under the µP scaling framework, allowing for effective learning rate transfer across different model sizes. The authors analyze the dynamics of sharpness and its correlation with learning rate transfer, contrasting it with the behavior observed under the Neural Tangent Kernel (NTK) framework, where such transfer doe

Author

Alexandru Meterez

Language

EN