Can I read The Impact of Cross-Lingual Adjustment of Contextual Word Representations on Zero-Shot Transfer on EtoBox?

The Impact of Cross-Lingual Adjustment of Contextual Word Representations on Zero-Shot Transfer by Efimov, Pavel; Boytsov, Leonid; Arslanova, Elena; Braslavski, Pavel is a scholarly article available to read on EtoBox.

What is The Impact of Cross-Lingual Adjustment of Contextual Word Representations on Zero-Shot Transfer about?

Large multilingual language models such as mBERT or XLM-R enable zero-shot cross-lingual transfer in various IR and NLP tasks. Cao et al. (2020) proposed a data- and compute-efficient method for cross-lingual adjustment of mBERT that uses a small parallel corpus to make embeddings of related words across languages similar to each other. They showed it to be effective in NLI for five European languages. In contrast we experiment with a typologically diverse set of languages (Spanish, Russian, Vietnamese, and Hindi) and extend their original implementations to new tasks (XSR, NER, and QA) and an additional training regime (continual learning). Our study reproduced gains in NLI for four languages, showed improved NER, XSR, and cross-lingual QA results in three languages (though some cross-lingual QA gains were not statistically significant), while mono-lingual QA performance never improved and sometimes degraded. Analysis of distances between contextualized embeddings of related and unrelated words (across languages) showed that fine-tuning leads to "forgetting" some of the cross-lingual alignment information. Based on this observation, we further improved NLI performance using contin

Author

Efimov, Pavel; Boytsov, Leonid; Arslanova, Elena; Braslavski, Pavel

Published

2022

Language

EN