Can I read Logic.py: Bridging the Gap between LLMs and Constraint Solvers on EtoBox?

Logic.py: Bridging the Gap between LLMs and Constraint Solvers by Kesseli, Pascal; O'Hearn, Peter; Cabral, Ricardo Silveira is a scholarly article available to read on EtoBox.

What is Logic.py: Bridging the Gap between LLMs and Constraint Solvers about?

We present a novel approach to formalise and solve search-based problems using large language models, which significantly improves upon previous state-of-the-art results. We demonstrate the efficacy of this approach on the logic puzzles benchmark ZebraLogicBench. Instead of letting the LLM attempt to directly solve the puzzles, our method prompts the model to formalise the problem in a logic-focused domain-specific language (DSL) called Logic.py. This formalised representation is then solved using a constraint solver, leveraging the strengths of both the language model and the solver. Our approach achieves a remarkable 65% absolute improvement over the baseline performance of Llama 3.1 70B on ZebraLogicBench, setting a new state-of-the-art with an accuracy of over 90%. This significant advancement demonstrates the potential of combining language models with domain-specific languages and auxiliary tools on traditionally challenging tasks for LLMs.

Author

Kesseli, Pascal; O'Hearn, Peter; Cabral, Ricardo Silveira

Published

2025

Language

EN