The paper introduces a reference‑free proxy‑judge framework for autoformalization, separating correctness into grounding, statement faithfulness, coverage, and inference. It presents three theoretical results linking proxy error rates to pass‑but‑wrong rates, the impact of false rejections on refinement, and a decomposition of final pass rates. The framework is instantiated with eight proxies and evaluated on ProofNet, miniF2F, e‑SNLI, and ProntoQA, showing that the predictions hold and identifying failure routes.
By Lei Xu, Xin Quan, Andr\'e Freitas
arXiv:2609.21190v1 Announce Type: cross
Abstract: Ensuring the correctness of LLM-generated code is a core challenge for modern software engineering. Benchmarks for agentic code generation check corr...
By George Ma, Benjamin Mikek, Haoyu Li, Ferhat Erata, Yuhao Zhang, Zeren Shui, Behrooz Omidvar Tehrani, Jun Huan, Murali Krishna Ramanathan, Somayeh Sojoudi, Hao Zhou, Anoop Deoras
The paper presents a unified dataset of 350 Lean tasks to evaluate specification generation (SpecGen) for verifiable code, combining metrics of formal validity, reference similarity, equivalence, and behavioural adequacy. It highlights that restricting similarity measures to a subset of tasks can shift rankings, and shows that a perfect postcondition score can still miss critical input coverage, underscoring the need for separate feedback on input and output constraints.
By Srijith Nair, Aditya Vempaty, Jia Liu, Ashish Jagmohan
arXiv:2607. 12650v1 Announce Type: cross Abstract: Tool access alone does not make LLM empirical reasoning governable: accepted outputs need not descend from attested evidence, and accepted deductions need not hold up under formal scrutiny.
By Junyu Ren
The paper introduces GUARD, a neuro‑symbolic system that autoformalizes argumentative material by completing missing premises (guards) before formal verification. It uses large language models to generate candidate guards, Isabelle/HOL to verify them, and a contrastive test to ensure the proof depends on the original premises and does not over‑generalize. Experiments on Debatepedia and ARCT show that GUARD improves verified‑faithful scores by over 30 points and reduces leakage by about 20 points compared to prior LLM‑driven theorem proving methods.
By Xin Quan, Reto Gubelmann, Andr\'e Freitas
arXiv:2607. 19407v1 Announce Type: new Abstract: Formal theorem proving has emerged as a frontier challenge for machine learning, yet the ecosystem is fragmented: proofs remain siloed across incompatible systems, limiting both training data for learning-based provers and the portability of verified results.
By Jiayi Wu, Robert Joseph George, Anima Anandkumar
arXiv:2606. 16541v1 Announce Type: new Abstract: Autoformalization, translating natural-language mathematics into formal proof assistants, is bottlenecked not by translation fluency but by \emph{faithfulness}: a formal statement can typecheck and be provable, yet still encode a different theorem than the source intended.
By Noor Islam S. Mohammad, Tamim Sheikh
FaithSieve is a Lean‑assisted framework that fine‑grains natural‑language mathematical proofs into local reasoning units, extracts typed proof obligations, and verifies them with formal evidence gated by semantic alignment. It introduces two expert‑verified datasets—ProofLoc‑Olympiad and ProofLoc‑University—to benchmark first‑error localization. On these benchmarks, FaithSieve outperforms direct‑judging baselines, achieving 81.43% and 84.5% exact first‑error accuracy respectively.
By Ziyu Wang, Qiming Dai, Yishan Wu, Zaiwen Wen
arXiv:2608. 15432v1 Announce Type: new Abstract: In formal verification, both the autoformalization of statements and automated proof search have been studied extensively.
By Tadd Mao, Tianjun Zhong, Dhruva Arekar, Yuming Feng, One An, Jiani Huang, Xujie Si, Ziyang Li
arXiv:2608. 20153v1 Announce Type: new Abstract: Large language models (LLMs) have shown growing potential for automated theoretical computer science (TCS) research, yet existing benchmarks remain far from realistic research settings.
By Dingzirui Wang, Xuanliang Zhang, Keyan Xu, Qingfu Zhu, Wanxiang Che
arXiv:2606. 29493v1 Announce Type: new Abstract: Benchmarks for LLM-assisted theorem proving in Lean are often treated as intrinsically reliable because every solved instance comes with a machine-checked proof.
By Pawan Sasanka Ammanamanchi, Siddharth Bhat, Stella Biderman
arXiv:2606. 31002v1 Announce Type: new Abstract: Theorem-proving benchmarks evaluate proof search against fixed formal statements, but natural-language-to-Lean formalization must generate the formal statement itself.
By Ke Zhang, Patricio Gallardo Candela, Sudhir Murthy, Yi Xie, Zhi Wang, Maziar Raissi