Can I read Loss-Free Balancing for MoE Models on EtoBox?

Loss-Free Balancing for MoE Models by twwang.ntu is a document available to read on EtoBox.

What is Loss-Free Balancing for MoE Models about?

The document presents a novel load balancing strategy called Loss-Free Balancing for Mixture-of-Experts (MoE) models, which avoids the use of auxiliary loss to maintain expert load balance without introducing interference gradients. This method dynamically adjusts expert-wise biases based on recent loads, ensuring a balanced distribution of expert utilization while enhancing model performance. Experimental results demonstrate that Loss-Free Balancing outperforms traditional auxiliary-loss-controlled methods

Author

twwang.ntu

Language

EN