Can I read Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs on EtoBox?

Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs by Sap, Maarten; LeBras, Ronan; Fried, Daniel; Choi, Yejin is a scholarly article available to read on EtoBox.

What is Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs about?

Social intelligence and Theory of Mind (ToM), i.e., the ability to reason about the different mental states, intents, and reactions of all people involved, allow humans to effectively navigate and understand everyday social interactions. As NLP systems are used in increasingly complex social situations, their ability to grasp social dynamics becomes crucial. In this work, we examine the open question of social intelligence and Theory of Mind in modern NLP systems from an empirical and theory-based perspective. We show that one of today's largest language models (GPT-3; Brown et al., 2020) lacks this kind of social intelligence out-of-the box, using two tasks: SocialIQa (Sap et al., 2019), which measures models' ability to understand intents and reactions of participants of social interactions, and ToMi (Le et al., 2019), which measures whether models can infer mental states and realities of participants of situations. Our results show that models struggle substantially at these Theory of Mind tasks, with well-below-human accuracies of 55% and 60% on SocialIQa and ToMi, respectively. To conclude, we draw on theories from pragmatics to contextualize this shortcoming of large language

Author

Sap, Maarten; LeBras, Ronan; Fried, Daniel; Choi, Yejin

Published

2022

Language

EN