Can I read ScheduleFree+ Paper (Learning Rate) on EtoBox?

ScheduleFree+ Paper (Learning Rate) by badri2006nathan is a document available to read on EtoBox.

What is ScheduleFree+ Paper (Learning Rate) about?

The document presents ScheduleFree+, a learning-rate-free and schedule-free method for training large language models, which significantly outperforms traditional Warmup-Stable-Decay schedules by 31% at 1000 tokens per parameter. It highlights the importance of modifications such as reintroducing inner momentum and using a variant of the Polyak step size to enhance performance, particularly for long-duration training. The findings suggest that Schedule-Free Learning is a promising approach for achieving sta

Author

badri2006nathan

Language

EN