Can I read Causal Reasoning and Large Language Models: Opening a New Frontier for Causality on EtoBox?

Causal Reasoning and Large Language Models: Opening a New Frontier for Causality by Kıcıman, Emre; Ness, Robert; Sharma, Amit; Tan, Chenhao is a scholarly article available to read on EtoBox.

What is Causal Reasoning and Large Language Models: Opening a New Frontier for Causality about?

The causal capabilities of large language models (LLMs) are a matter of significant debate, with critical implications for the use of LLMs in societally impactful domains such as medicine, science, law, and policy. We conduct a "behavorial" study of LLMs to benchmark their capability in generating causal arguments. Across a wide range of tasks, we find that LLMs can generate text corresponding to correct causal arguments with high probability, surpassing the best-performing existing methods. Algorithms based on GPT-3.5 and 4 outperform existing algorithms on a pairwise causal discovery task (97%, 13 points gain), counterfactual reasoning task (92%, 20 points gain) and event causality (86% accuracy in determining necessary and sufficient causes in vignettes). We perform robustness checks across tasks and show that the capabilities cannot be explained by dataset memorization alone, especially since LLMs generalize to novel datasets that were created after the training cutoff date. That said, LLMs exhibit unpredictable failure modes, and we discuss the kinds of errors that may be improved and what are the fundamental limits of LLM-based answers. Overall, by operating on the text metad

Author

Kıcıman, Emre; Ness, Robert; Sharma, Amit; Tan, Chenhao

Published

2023

Language

EN