Can I read Evaluating Transformer's Ability to Learn Mildly Context-Sensitive Languages on EtoBox?

Evaluating Transformer's Ability to Learn Mildly Context-Sensitive Languages by Wang, Shunjie; Steinert-Threlkeld, Shane is a scholarly article available to read on EtoBox.

What is Evaluating Transformer's Ability to Learn Mildly Context-Sensitive Languages about?

Despite the fact that Transformers perform well in NLP tasks, recent studies suggest that self-attention is theoretically limited in learning even some regular and context-free languages. These findings motivated us to think about their implications in modeling natural language, which is hypothesized to be mildly context-sensitive. We test the Transformer's ability to learn mildly context-sensitive languages of varying complexities, and find that they generalize well to unseen in-distribution data, but their ability to extrapolate to longer strings is worse than that of LSTMs. Our analyses show that the learned self-attention patterns and representations modeled dependency relations and demonstrated counting behavior, which may have helped the models solve the languages.

Author

Wang, Shunjie; Steinert-Threlkeld, Shane

Published

2023

Language

EN