Can I read GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models on EtoBox?

GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models by Mirzadeh, Iman; Alizadeh, Keivan; Shahrokhi, Hooman; Tuzel, Oncel; Bengio, Samy; Farajtabar, Mehrdad is a scholarly article available to read on EtoBox.

What is GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models about?

Recent advancements in Large Language Models (LLMs) have sparked interest in their formal reasoning capabilities, particularly in mathematics. The GSM8K benchmark is widely used to assess the mathematical reasoning of models on grade-school-level questions. While the performance of LLMs on GSM8K has significantly improved in recent years, it remains unclear whether their mathematical reasoning capabilities have genuinely advanced, raising questions about the reliability of the reported metrics. To address these concerns, we conduct a large-scale study on several SOTA open and closed models. To overcome the limitations of existing evaluations, we introduce GSM-Symbolic, an improved benchmark created from symbolic templates that allow for the generation of a diverse set of questions. GSM-Symbolic enables more controllable evaluations, providing key insights and more reliable metrics for measuring the reasoning capabilities of models.Our findings reveal that LLMs exhibit noticeable variance when responding to different instantiations of the same question. Specifically, the performance of all models declines when only the numerical values in the question are altered in the GSM-Symbolic

Author

Mirzadeh, Iman; Alizadeh, Keivan; Shahrokhi, Hooman; Tuzel, Oncel; Bengio, Samy; Farajtabar, Mehrdad

Published

2024

Language

EN