AdvancedMathBench is a new benchmark suite that evaluates large language models on advanced mathematical proof generation and verification. It includes ProverBench, with 245 problems from undergraduate to doctoral qualifying‑exam levels, and VerifierBench, which tests models’ ability to judge proof validity using 888 expert‑annotated trajectories. The suite features an automatic verification pipeline trained on expert data, and results show that even state‑of‑the‑art models perform poorly, highlighting a gap between generation and verification skills.
By Lingkai Kong, Zijian Wu, Yuzhe Gu, Haiteng Zhao, Zhouqi Hua, Wenyong Huang, Shuang Sun, Zhicheng Xiong, Xiaotian Zhang, Shuya Zhao, Yan Wang, Disheng Xu, Wenwei Zhang, Kai Chen
Large language models (LLMs) have achieved remarkable performance on high-school and olympiad-style mathematics, yet their capabilities on advanced mathematics remain poorly understood. Existing benchmarks, however, fall short in both scope and evaluation granularity: they provide limited disciplinary coverage and often rely on final-answer correctness or coarse judgments, leaving the validity of the reasoning process inadequately assessed.
arXiv:2603. 15510v2 Announce Type: replace Abstract: The synthesis of inductive loop invariants remains a critical bottleneck in automated program verification.
By Ido Pinto, Yizhak Yisrael Elboher, Haoze Wu, Nina Narodytska, Guy Katz
arXiv:2608. 19009v2 Announce Type: replace Abstract: Large language models (LLMs) are increasingly paired with verifiers (step checkers, self-consistency filters, tool-based fact checkers, formal proof assistants) that claim to detect the model's errors.
By Yajie Yin
arXiv:2603. 18334v2 Announce Type: replace-cross Abstract: As Large Language Models (LLMs) increasingly assist secure software development, their ability to meet the rigorous demands of Rust program verification remains unclear.
By Zichen Xie, Wenxi Wang
FaithSieve is a Lean‑assisted framework that fine‑grains natural‑language mathematical proofs into local reasoning units, extracts typed proof obligations, and verifies them with formal evidence gated by semantic alignment. It introduces two expert‑verified datasets—ProofLoc‑Olympiad and ProofLoc‑University—to benchmark first‑error localization. On these benchmarks, FaithSieve outperforms direct‑judging baselines, achieving 81.43% and 84.5% exact first‑error accuracy respectively.
By Ziyu Wang, Qiming Dai, Yishan Wu, Zaiwen Wen
The paper evaluates how robust large language models are at generating SystemVerilog Assertions (SVA) when the underlying RTL code undergoes semantics‑preserving transformations such as operand reordering, identifier renaming, and redundant parenthesization. Using a curated dataset and two open‑source models (Qwen2.5‑Coder‑7B and DeepSeek‑Coder‑V2‑Lite), the authors find that 9.7%–27.0% of behaviors that were correct on the original RTL become incorrect after transformation, revealing significant instability that aggregate accuracy metrics can hide.
By FNU Aditi
arXiv:2606. 15258v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly capable of mathematical problem solving and can even assist with research-level proofs, yet we still lack a scalable and reproducible way to measure step-level reasoning in long proofs across diverse sources.
By Jierui Zhang, Siyuan Tan, Xinhang Li, Longzhuangzhi Lin, Dailin Li, Chengfeng Gu, Xinping Li, Yaxian Hao, Shengjia Liang, Yuxiang Ren, Wenhao Liu
The paper introduces SOVER, a framework that uses Large Language Models (LLMs) to extract semantic mappings between optimization reformulations and then formally verifies these mappings with SMT solvers. Z3 is employed to check domain cross-feasibility and objective-order preservation for mixed-integer linear problems, while dReal handles tolerance-aware feasibility and ε-argmin checks for continuous nonlinear problems. The authors also present NLEquiv-150, a benchmark of 150 nonlinear reformulation pairs, and report that SOVER correctly classifies 149 out of 150 pairs, including all 50 hard negatives, with the single error due to incomplete mapping extraction.
By Swapnil Bhattacharyya, Mayank Baranwal
The paper introduces OMVV, an online multi-verifier algorithm that maintains a pool of weak verifiers with varying costs and performance. It adaptively selects a verifier each round using an online score combiner and exponential-weights routing, providing distribution-free guarantees on false-accept and false-reject rates. Experiments on reasoning benchmarks show OMVV achieves higher accuracy at lower verification cost than any single fixed verifier across different budgets.
By Erfan Hajihashemi, Yanning Shen
The paper introduces DRAGO, a framework that repairs formal specifications generated by large language models (LLMs) using deterministic, formalism‑specific diagnosis instead of ungrounded LLM critiques. DRAGO selectively rewrites specifications only when the diagnosed error weight decreases, reducing regressions from 11.8% to 1.3% and improving answer accuracy by 12.9 percentage points over zero‑shot formalization and 8.7 percentage points over LLM‑diagnostic repair across five benchmarks. Additionally, the authors identified and corrected rendering defects in Planetarium that caused half of its floor‑tile split instances to be inconsistent with reference formalizations.
By Giovanni Vaccarino, Emanuele La Malfa, Dehai Min, Hanrong Zhang, Pier Luca Lanzi, Lu Cheng, Michael J. Wooldridge, Philip S. Yu, Sara Bernardini
arXiv:2602. 15983v3 Announce Type: replace-cross Abstract: Large language models (LLMs) can translate natural language into optimization code, but silent failures pose a critical risk: code that executes and returns solver-feasible solutions may encode semantically incorrect formulations---a feasibility--correctness gap reaching 90 percentage points on compositional problems.
By Junbo Jacob Lian, Yujun Sun, Huiling Chen, Chaoyu Zhang, Hanzhang Qin, Chung-Piaw Teo