Can I read Assessing Language Comprehension in Large Language Models Using Construction Grammar on EtoBox?

Assessing Language Comprehension in Large Language Models Using Construction Grammar by Scivetti, Wesley; Torgbi, Melissa; Blodgett, Austin; Shichman, Mollie; Hudson, Taylor; Bonial, Claire; Madabushi, Harish Tayyar is a scholarly article available to read on EtoBox.

What is Assessing Language Comprehension in Large Language Models Using Construction Grammar about?

Large Language Models, despite their significant capabilities, are known to fail in surprising and unpredictable ways. Evaluating their true `understanding' of language is particularly challenging due to the extensive web-scale data they are trained on. Therefore, we construct an evaluation to systematically assess natural language understanding (NLU) in LLMs by leveraging Construction Grammar (CxG), which provides insights into the meaning captured by linguistic elements known as constructions (Cxns). CxG is well-suited for this purpose because provides a theoretical basis to construct targeted evaluation sets. These datasets are carefully constructed to include examples which are unlikely to appear in pre-training data, yet intuitive and easy for humans to understand, enabling a more targeted and reliable assessment. Our experiments focus on downstream natural language inference and reasoning tasks by comparing LLMs' understanding of the underlying meanings communicated through 8 unique Cxns with that of humans. The results show that while LLMs demonstrate some knowledge of constructional information, even the latest models including GPT-o1 struggle with abstract meanings conveye

Author

Scivetti, Wesley; Torgbi, Melissa; Blodgett, Austin; Shichman, Mollie; Hudson, Taylor; Bonial, Claire; Madabushi, Harish Tayyar

Published

2025

Language

EN