Can I read Optimal Language Mixture for Llama-3 on EtoBox?

Optimal Language Mixture for Llama-3 by rizzhbb is a document available to read on EtoBox.

What is Optimal Language Mixture for Llama-3 about?

This paper investigates the Continual Pre-Training (CPT) of the Llama-3 70B model to enhance its performance in unfamiliar languages and specific domains, particularly focusing on the optimal selection of the Additional Language Mixture Ratio (ALMR) and Learning Rate (LR). Through extensive experiments, the authors establish a correlation between ALMR and LR, demonstrating significant improvements in the model

Author

rizzhbb

Language

EN