arXiv AI

Navier-Stokes lost in translation: Why Lean verification of AI autoformalisation does not guarantee correct natural language proofs

arXiv AI
Jun 16

The Faithfulness Gap: Certifying Semantic Equivalence Between Natural-Language and Formal Mathematical Statements

arXiv:2606. 16541v1 Announce Type: new Abstract: Autoformalization, translating natural-language mathematics into formal proof assistants, is bottlenecked not by translation fluency but by \emph{faithfulness}: a formal statement can typecheck and be provable, yet still encode a different theorem than the source intended.

By Noor Islam S. Mohammad, Tamim Sheikh
arXiv AI
Jun 2

Formally Solving Answer-Construction Problems in Lean

arXiv:2505. 18492v5 Announce Type: replace Abstract: Mathematical competition problems fall into two broad types: theorem proving, which asks for a proof of a given statement, and answer construction, which requires constructing a property-satifying object with proofs.

By Jialiang Sun, Yuzhi Tang, Ao Li, Chris J. Maddison, Kuldeep S. Meel
arXiv AI
Aug 28

FaithSieve: Fine-Grained Evaluation of Math Proofs with Faithful Formal Evidence

FaithSieve is a Lean‑assisted framework that fine‑grains natural‑language mathematical proofs into local reasoning units, extracts typed proof obligations, and verifies them with formal evidence gated by semantic alignment. It introduces two expert‑verified datasets—ProofLoc‑Olympiad and ProofLoc‑University—to benchmark first‑error localization. On these benchmarks, FaithSieve outperforms direct‑judging baselines, achieving 81.43% and 84.5% exact first‑error accuracy respectively.

By Ziyu Wang, Qiming Dai, Yishan Wu, Zaiwen Wen
arXiv Computation and Language
Sep 16

Autoformalizing Argumentative Material Inferences

The paper introduces GUARD, a neuro‑symbolic system that autoformalizes argumentative material by completing missing premises (guards) before formal verification. It uses large language models to generate candidate guards, Isabelle/HOL to verify them, and a contrastive test to ensure the proof depends on the original premises and does not over‑generalize. Experiments on Debatepedia and ARCT show that GUARD improves verified‑faithful scores by over 30 points and reduces leakage by about 20 points compared to prior LLM‑driven theorem proving methods.

By Xin Quan, Reto Gubelmann, Andr\'e Freitas
arXiv AI
Oct 26

PaperFormalizer: Author-in-the-Loop Autoformalization of Research Papers in Lean

PaperFormalizer is a Lean-based pipeline that assists authors in identifying and closing gaps in their research papers, converting them into Lean-verified versions. Starting from LaTeX, it transcribes the paper into a dependency graph, audits statements for faithfulness and correctness, and formalizes them in Lean using Mathlib and optional domain libraries. The process is author-in-the-loop, prompting the author whenever multiple interpretations exist and allowing them to accept, decline, or redirect proposed repairs, ultimately producing an annotated paper, a change report, and linked Lean code.

By Jiyuan Tan, Vasilis Syrgkanis, Jose Blanchet
arXiv Computation and Language
Aug 24

Beyond Gold Standards: Epistemic Ensemble of LLM Judges for Formal Mathematical Reasoning

The paper introduces an epistemically and formally grounded ensemble (EFG) of large language model judges to evaluate autoformalization tasks in formal mathematics. It defines four criteria—logical preservation, mathematical consistency, formal quality, and formal validity—to provide a transparent, multi‑granular assessment. Experiments show that this ensemble outperforms coarse‑grained models, offering a scalable and interpretable proxy for evaluating formal mathematical reasoning.

By Lan Zhang, Marco Valentino, Jordan Meadows, Andre Freitas