The paper argues that the term "verified" in LLM‑generated code is ambiguous, identifying three distinct failure surfaces: verifier acceptance (Surface A), syntactic specification binding (Surface B), and semantic specification fidelity (Surface C). Empirical tests on Python and Rust benchmarks show that high verifier acceptance does not guarantee binding to a specification, and even fully‑verified Rust outputs can fail hidden tests, revealing gaps between formal proofs and real‑world correctness.
arXiv:2602. 09464v2 Announce Type: replace-cross Abstract: Vericoding refers to the generation of formally verified code from rigorous specifications.
arXiv:2610.17358v1 Announce Type: cross Abstract: Formal verification proves that a program satisfies its specification, but says nothing about whether the specification captures what the author mean...
arXiv:2606. 13706v1 Announce Type: cross Abstract: We present HierSVA, an integrated suite that combines a pipeline, dataset, and benchmark for LLM-driven hierarchical hardware formal verification.
arXiv:2607. 12650v1 Announce Type: cross Abstract: Tool access alone does not make LLM empirical reasoning governable: accepted outputs need not descend from attested evidence, and accepted deductions need not hold up under formal scrutiny.
arXiv:2609.21190v1 Announce Type: cross Abstract: Ensuring the correctness of LLM-generated code is a core challenge for modern software engineering. Benchmarks for agentic code generation check corr...
SemLoc transforms free‑form reasoning from large language models into executable semantic constraints that are attached to specific code anchors such as variable definitions, branches, loops, or returns. These constraints are inserted as runtime checks, and the resulting semantic violation spectrum is used to guide counterfactual verification and patch generation. On the QuixBugs and new SemBugs datasets, SemLoc’s constraints achieve high ranking of faulty lines and a high proportion of verifiable, correct patches, matching the accuracy of the underlying LLMs while outperforming traditional spectrum‑based methods.
The paper introduces RefactorProof, a benchmark that tests whether proofs generated for code remain valid after program refactoring that preserves behavior. Using 458 Lean-certified variants derived from 185 VERINA tasks, the study finds that structural refactorings expose brittleness in proofs, with task-normalized structural PSR ranging from 4.3% for supplied references to 13.3–26.8% for specialized provers. Despite higher failure rates, specialized prover proofs (DeepSeek-Prover-V2-7B, Goedel-Prover-V2-32B, Goedel-Prover-V2-8B) show significantly higher unchanged-proof survival compared to reference artifacts, and many failures can be repaired with minimal edits.
SemVerBench is a benchmark that evaluates how well large language models (LLMs) understand and apply version-constraint resolution semantics, such as determining whether a version satisfies constraints like ^1.2.3 or >=2.0. The study finds that many models struggle with certain corner cases, with GPT‑5.1 performing poorly while Claude and Opus perform much better. The authors suggest that the failures stem from an activation/application gap rather than a lack of knowledge, and recommend that coding agents delegate version resolution to a dedicated resolver tool.
arXiv:2606. 07316v2 Announce Type: replace-cross Abstract: Can a committee of LLM agents reach agreement that is certifiable at the level of meaning, not only at the level of a label?
arXiv:2607. 10291v1 Announce Type: cross Abstract: Software evolves continuously, yet ensuring that a patch preserves intended behavior without re-verifying an entire codebase remains difficult.
The paper presents a unified dataset of 350 Lean tasks to evaluate specification generation (SpecGen) for verifiable code, combining metrics of formal validity, reference similarity, equivalence, and behavioural adequacy. It highlights that restricting similarity measures to a subset of tasks can shift rankings, and shows that a perfect postcondition score can still miss critical input coverage, underscoring the need for separate feedback on input and output constraints.
The paper compares a non‑generative verifier (Jev) with a frontier LLM (Astra) on 495 human‑labelled claims from LLM‑AggreFact. Both models achieve statistically indistinguishable accuracy, yet Jev is 187‑fold cheaper and admits more unsupported claims. A confidence‑gated cascade that escalates uncertain claims improves cost‑efficiency, though the benefit comes mainly from the escalation share rather than the probability estimates.