ProofJudge is an agentic large language model that evaluates the quality of formal proofs in Lean 4 beyond mere correctness. It scores proofs on five dimensions—library leverage, automation fit, structural clarity, statement quality, and Mathlib conventions—using tool access to the relevant repository commit. The system was tested on 218 Mathlib declarations, achieving alignment with human reviewers between 63.5% and 80.8% and releasing its harness, dataset, and traces for open research.
By Shane Caldwell
arXiv:2602.18307v2 Announce Type: replace-cross
Abstract: Large language models have achieved striking results in interactive theorem proving, particularly in Lean. However, most benchmarks for LLM-b...
By Yutong Xin, Qiaochu Chen, Greg Durrett, I\c{s}il Dillig
FaithSieve is a Lean‑assisted framework that fine‑grains natural‑language mathematical proofs into local reasoning units, extracts typed proof obligations, and verifies them with formal evidence gated by semantic alignment. It introduces two expert‑verified datasets—ProofLoc‑Olympiad and ProofLoc‑University—to benchmark first‑error localization. On these benchmarks, FaithSieve outperforms direct‑judging baselines, achieving 81.43% and 84.5% exact first‑error accuracy respectively.
By Ziyu Wang, Qiming Dai, Yishan Wu, Zaiwen Wen
arXiv:2606. 29493v1 Announce Type: new Abstract: Benchmarks for LLM-assisted theorem proving in Lean are often treated as intrinsically reliable because every solved instance comes with a machine-checked proof.
By Pawan Sasanka Ammanamanchi, Siddharth Bhat, Stella Biderman
arXiv:2608. 00004v1 Announce Type: cross Abstract: Grading natural-language mathematical proofs is a recurring cost in evaluating math-reasoning systems, and frontier LLM judges are expensive.
By Benjamin Grayzel
arXiv:2610.17358v1 Announce Type: cross
Abstract: Formal verification proves that a program satisfies its specification, but says nothing about whether the specification captures what the author mean...
By Martin Amster, One An, Holly Baker, Fernanda Graciolli, Simon Henniger, Nada Amin
Large language models produce outputs presented as discoveries - new proofs, conjectures, or molecules. Whether such an output that appears creative is truly original and effective is hard to establis...
arXiv:2606. 31002v1 Announce Type: new Abstract: Theorem-proving benchmarks evaluate proof search against fixed formal statements, but natural-language-to-Lean formalization must generate the formal statement itself.
By Ke Zhang, Patricio Gallardo Candela, Sudhir Murthy, Yi Xie, Zhi Wang, Maziar Raissi
KAPPA is an open‑source toolkit that uses language‑model agents to translate LaTeX proofs from statistical papers into Lean 4, where each numbered result is verified step by step against the original text. The system assigns one of four verdicts—verified unconditionally, verified conditionally, issue recorded, or unfinished—to every result, and it can run unattended, re‑check only changed sections, and report resource usage. In tests on four example papers and three published statistical papers, KAPPA detected the known errors and additional issues that were later confirmed by hand, while a direct language‑model approach without the kit produced fewer correct verdicts and sometimes over‑claimed proofs.
By Zihua She, Xiao Wang
LLMs have recently achieved strong results on formal proving benchmarks. However, existing evaluations remain heavily concentrated on competition-style problems and often fail to capture how models behave on longer, more dependency-rich mathematical developments.
We introduce a novel method for verifying Lean formalizations that goes beyond LLM-judged faithfulness. Unlike most verification methods, which rely on LLMs or trained models, our method combines LLM-...
FORALL-LEAN-AGENT is a frontend‑agnostic framework that enhances auditable reasoning in formal mathematics and software verification. It integrates isolated workspaces, Lean tools, and fresh review processes—including statement comparison, axiom audits, and independent proof checking—to bind verification evidence and reviewer decisions to the same candidate artifact, ensuring traceable acceptance. Evaluations on VeriSoftBench, PutnamBench, and the Lean Eval track show that the framework improves benchmark success rates and reduces costs while providing evidence beyond aggregate solve counts.
By Naing Oo Lwin