Can I read From English-Centric to Effective Bilingual: LLMs with Custom Tokenizers for Underrepresented Languages on EtoBox?

From English-Centric to Effective Bilingual: LLMs with Custom Tokenizers for Underrepresented Languages by Kiulian, Artur; Polishko, Anton; Khandoga, Mykola; Kostiuk, Yevhen; Gabrielli, Guillermo; Gagała, Łukasz; Zaraket, Fadi; Obaida, Qusai Abu; Garud, Hrishikesh; Mak, Wendy Wing Yee; Chaplynskyi, Dmytro; Amor, Selma Belhadj; Peradze, Grigol is a scholarly article available to read on EtoBox.

What is From English-Centric to Effective Bilingual: LLMs with Custom Tokenizers for Underrepresented Languages about?

In this paper, we propose a model-agnostic cost-effective approach to developing bilingual base large language models (LLMs) to support English and any target language. The method includes vocabulary expansion, initialization of new embeddings, model training and evaluation. We performed our experiments with three languages, each using a non-Latin script - Ukrainian, Arabic, and Georgian. Our approach demonstrates improved language performance while reducing computational costs. It mitigates the disproportionate penalization of underrepresented languages, promoting fairness and minimizing adverse phenomena such as code-switching and broken grammar. Additionally, we introduce new metrics to evaluate language quality, revealing that vocabulary size significantly impacts the quality of generated text.

Author

Kiulian, Artur; Polishko, Anton; Khandoga, Mykola; Kostiuk, Yevhen; Gabrielli, Guillermo; Gagała, Łukasz; Zaraket, Fadi; Obaida, Qusai Abu; Garud, Hrishikesh; Mak, Wendy Wing Yee; Chaplynskyi, Dmytro; Amor, Selma Belhadj; Peradze, Grigol

Published

2024

Language

EN