Can I read Continual Pre-training of MoEs: How robust is your router? on EtoBox?

Continual Pre-training of MoEs: How robust is your router? by Thérien, Benjamin; Joseph, Charles-Étienne; Sarwar, Zain; Panda, Ashwinee; Das, Anirban; Zhang, Shi-Xiong; Rawls, Stephen; Sahu, Sambit; Belilovsky, Eugene; Rish, Irina is a scholarly article available to read on EtoBox.

What is Continual Pre-training of MoEs: How robust is your router? about?

Sparsely-activated Mixture of Experts (MoE) transformers are promising architectures for foundation models. Compared to dense transformers that require the same amount of floating point operations (FLOPs) per forward pass, MoEs benefit from improved sample efficiency at training time and achieve much stronger performance. Many closed-source and open-source frontier language models have thus adopted an MoE architecture. Naturally, practitioners will want to extend the capabilities of these models with large amounts of newly collected data without completely re-training them. Prior work has shown that a simple combination of replay and learning rate re-warming and re-decaying can enable the continual pre-training (CPT) of dense decoder-only transformers with minimal performance degradation compared to full re-training. In the case of decoder-only MoE transformers, however, it is unclear how the routing algorithm will impact continual pre-training performance: 1) do the MoE transformer's routers exacerbate forgetting relative to a dense model?; 2) do the routers maintain a balanced load on previous distributions after CPT?; 3) are the same strategies applied to dense models sufficient

Author

Thérien, Benjamin; Joseph, Charles-Étienne; Sarwar, Zain; Panda, Ashwinee; Das, Anirban; Zhang, Shi-Xiong; Rawls, Stephen; Sahu, Sambit; Belilovsky, Eugene; Rish, Irina

Published

2025

Language

EN