arXiv AI November 5, 2026 By Aditya Vempaty, Himaneesh Sompalle, Manan Tayal, Mowafak Allaham, Prasenjit Dey, Ashish Jagmohan, Ravi Kokku

Verified but Vacuous: Three Failure Surfaces of LLM-Driven Verified Code Generation

Read the original on arXiv AI →

The paper argues that the term "verified" in LLM‑generated code is ambiguous, identifying three distinct failure surfaces: verifier acceptance (Surface A), syntactic specification binding (Surface B), and semantic specification fidelity (Surface C). Empirical tests on Python and Rust benchmarks show that high verifier acceptance does not guarantee binding to a specification, and even fully‑verified Rust outputs can fail hidden tests, revealing gaps between formal proofs and real‑world correctness.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI

Jun 4

AlgoVeri: An Aligned Benchmark for Verified Code Generation on Classical Algorithms

arXiv:2602. 09464v2 Announce Type: replace-cross Abstract: Vericoding refers to the generation of formally verified code from rigorous specifications.

By Haoyu Zhao, Ziran Yang, Jiawei Li, Deyuan He, Zenan Li, Chi Jin, Venugopal V. Veeravalli, Aarti Gupta, Sanjeev Arora

llms benchmarks

arXiv AI

Oct 15

claimcheck: Measuring and Improving Specification Faithfulness

arXiv:2610.17358v1 Announce Type: cross Abstract: Formal verification proves that a program satisfies its specification, but says nothing about whether the specification captures what the author mean...

By Martin Amster, One An, Holly Baker, Fernanda Graciolli, Simon Henniger, Nada Amin

llms benchmarks

arXiv AI

Jun 15

HierSVA: A Data Synthesis Pipeline, Dataset, and Benchmark for LLM-Driven Hierarchical Hardware Formal Verification

arXiv:2606. 13706v1 Announce Type: cross Abstract: We present HierSVA, an integrated suite that combines a pipeline, dataset, and benchmark for LLM-driven hierarchical hardware formal verification.

By Maohua Nie, Jiang Zhu, Jingqun Zhang, Zhichen Zeng, Jiayi Wang, Sibo Zhang, Jialin Wang, C. -J. Richard Shi

llms agents benchmarks

arXiv AI

Jul 15

Evidence-Grounded Verified Agentic Reasoning: A Path Toward Eliminating LLM Hallucination in Empirical Inference via Tool-Attested Kernel Proofs

arXiv:2607. 12650v1 Announce Type: cross Abstract: Tool access alone does not make LLM empirical reasoning governable: accepted outputs need not descend from attested evidence, and accepted deductions need not hold up under formal scrutiny.

By Junyu Ren

llms agents safety

arXiv AI

Sep 21

SWE-Proof: Can Language Models Resolve Real-World Issues with Machine-Checked Proofs?

arXiv:2609.21190v1 Announce Type: cross Abstract: Ensuring the correctness of LLM-generated code is a core challenge for modern software engineering. Benchmarks for agentic code generation check corr...

By George Ma, Benjamin Mikek, Haoyu Li, Ferhat Erata, Yuhao Zhang, Zeren Shui, Behrooz Omidvar Tehrani, Jun Huan, Murali Krishna Ramanathan, Somayeh Sojoudi, Hao Zhou, Anoop Deoras

llms agents benchmarks safety

arXiv AI

Oct 28

SemLoc: Structured Grounding of Free-Form LLM Reasoning for Fault Localization

SemLoc transforms free‑form reasoning from large language models into executable semantic constraints that are attached to specific code anchors such as variable definitions, branches, loops, or returns. These constraints are inserted as runtime checks, and the resulting semantic violation spectrum is used to guide counterfactual verification and patch generation. On the QuixBugs and new SemBugs datasets, SemLoc’s constraints achieve high ranking of faulty lines and a high proportion of verifiable, correct patches, matching the accuracy of the underlying LLMs while outperforming traditional spectrum‑based methods.

By Zhaorui Yang, Haichao Zhu, Qian Zhang, Rajiv Gupta, Ashish Kundu

llms

One email a morning, machine-written

Related stories

AlgoVeri: An Aligned Benchmark for Verified Code Generation on Classical Algorithms

claimcheck: Measuring and Improving Specification Faithfulness

HierSVA: A Data Synthesis Pipeline, Dataset, and Benchmark for LLM-Driven Hierarchical Hardware Formal Verification

Evidence-Grounded Verified Agentic Reasoning: A Path Toward Eliminating LLM Hallucination in Empirical Inference via Tool-Attested Kernel Proofs

SWE-Proof: Can Language Models Resolve Real-World Issues with Machine-Checked Proofs?

SemLoc: Structured Grounding of Free-Form LLM Reasoning for Fault Localization