Skip to content

Opening book details…

Can I read Targeted Multilingual Adaptation for Low-resource Language Families on EtoBox?

Targeted Multilingual Adaptation for Low-resource Language Families by Downey, C. M.; Blevins, Terra; Serai, Dhwani; Parikh, Dwija; Steinert-Threlkeld, Shane is a scholarly article available to read on EtoBox.

What is Targeted Multilingual Adaptation for Low-resource Language Families about?

The "massively-multilingual" training of multilingual models is known to limit their utility in any one language, and they perform particularly poorly on low-resource languages. However, there is evidence that low-resource languages can benefit from targeted multilinguality, where the model is trained on closely related languages. To test this approach more rigorously, we systematically study best practices for adapting a pre-trained model to a language family. Focusing on the Uralic family as a test case, we adapt XLM-R under various configurations to model 15 languages; we then evaluate the performance of each experimental setting on two downstream tasks and 11 evaluation languages. Our adapted models significantly outperform mono- and multilingual baselines. Furthermore, a regression analysis of hyperparameter effects reveals that adapted vocabulary size is relatively unimportant for low-resource languages, and that low-resource languages can be aggressively up-sampled during training at little detriment to performance in high-resource languages. These results introduce new best practices for performing language adaptation in a targeted setting.

Author
Downey, C. M.; Blevins, Terra; Serai, Dhwani; Parikh, Dwija; Steinert-Threlkeld, Shane
Published
2024
Language
EN

More by Downey, C. M.; Blevins, Terra; Serai, Dhwani; Parikh, Dwija; Steinert-Threlkeld, Shane

Browse all works by Downey, C. M.; Blevins, Terra; Serai, Dhwani; Parikh, Dwija; Steinert-Threlkeld, Shane