arXiv AI
Sep 2

RePro: Proof-Verified Benchmark Rewriting for Reliable Evaluation of LLM Mathematical Problem Solving

RePro is a framework that rewrites benchmark problems for large language models (LLMs) in mathematical problem solving, ensuring that the rewritten problems and their answers are valid and correct through Lean-verified proofs. It integrates Lean-oriented neural automated theorem provers (ATPs) to regenerate answers, achieving 100% well-definedness, feasibility, and answer correctness on GSM8K and MATH datasets. Experiments show that models’ performance drops on these proof‑verified rewritten benchmarks, indicating sensitivity to surface‑level and structural variations and potential memorization effects.

By Xiyuan Zhou, Zhuoqi Li, Xinlei Wang, Yirui He, Yuhao Wu, Yuheng Cheng, Yan Xu, Junhua Zhao, Jinjin Gu
arXiv Machine Learning
Oct 16

Learning Lean Feedback for Efficient Proof-Candidate Evaluation on GPUs

The paper presents a neural evaluator that predicts Lean 4 theorem‑proving verdicts and compiler feedback, trained on 4.84 million labelled programs. A two‑stage approach—first screening candidates with a validity classifier, then scoring positives with a feedback model—achieves 87.8 % accuracy on a 68,932‑program benchmark and processes 3.5 candidates per second on an H100 GPU, outperforming a 64‑worker Lean baseline. Integrating this evaluator into a hybrid reward system reduces GRPO training steps by about 31 % while maintaining comparable miniF2F pass rates.

By Lazar Milikic, Etienne Bamas, Emmanuel Abbe, Viktor Kun\v{c}ak
arXiv AI
Jun 12

Pythagoras-Prover: Advancing Efficient Formal Proving via Augmented Lean Formalisation

arXiv:2606. 12594v1 Announce Type: new Abstract: Modern Lean theorem provers achieve strong performance only with substantial training and inference compute, driven in part by scarce verified proof data and the long reasoning traces of formal proof search, making both supervised fine-tuning (SFT) and sampling expensive.

By Joshua Ong Jun Leang, Zheng Zhao, Mihaela C\u{a}t\u{a}lina Stoian, Qiyuan Xu, Haonan Li, Wenda Li, Shay B. Cohen, Eleonora Giunchiglia