arXiv Machine Learning
VeriSoftBench: Repository-Scale Formal Verification Benchmarks for Lean
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
Hugging Face Trending Papers
Jun 8
TheoremBench: Evaluating LLMs on Theorem Proving in Formal Mathematics
LLMs have recently achieved strong results on formal proving benchmarks. However, existing evaluations remain heavily concentrated on competition-style problems and often fail to capture how models behave on longer, more dependency-rich mathematical developments.
arXiv AI
Jun 9
TheoremBench: Evaluating LLMs on Theorem Proving in Formal Mathematics
arXiv:2606. 09450v1 Announce Type: new Abstract: LLMs have recently achieved strong results on formal proving benchmarks.
arXiv AI
Aug 24
ProofJudge: Tool-Grounded LLM Evaluation of Formal Proof Quality in Mathlib
ProofJudge is an agentic large language model that evaluates the quality of formal proofs in Lean 4 beyond mere correctness. It scores proofs on five dimensions—library leverage, automation fit, structural clarity, statement quality, and Mathlib conventions—using tool access to the relevant repository commit. The system was tested on 218 Mathlib declarations, achieving alignment with human reviewers between 63.5% and 80.8% and releasing its harness, dataset, and traces for open research.
arXiv AI
Jun 16
SorryDB: Can AI Provers Complete Real-World Lean Theorems?
arXiv:2603. 02668v2 Announce Type: replace Abstract: We present SorryDB, a dynamically-updating benchmark of open Lean tasks drawn from 78 real world formalization projects on GitHub.
Hugging Face Trending Papers
Aug 13
Vero: Can AI Agents Build Formally Verified Software Repositories?
AI agents are increasingly used for programming, but do not provide any guarantee on the correctness of generated code. Verified code generation, in which an agent produces both an implementation and a machine-checked proof of its specification, offers a stronger path toward trustworthy AI-generated software.
Hugging Face Trending Papers
Aug 20
ProofJudge: Tool-Grounded LLM Evaluation of Formal Proof Quality in Mathlib
Formal proofs in Lean 4 that pass the kernel's type checker can nonetheless vary widely in quality. We introduce ProofJudge, an agentic LLM-as-judge system that scores formal proof quality along five...