arXiv AI
Spec-Harness: Assessing and Guiding LLM-Synthesized Formal Specifications Beyond Verifiability
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Sep 11
Spec-Harness: Measuring and Improving Behavioral Adequacy of LLM-Synthesized Formal Specifications
Spec‑Harness evaluates how well large language models (LLMs) synthesize Java Modeling Language (JML) specifications by measuring behavioral adequacy across precondition and postcondition correctness and completeness. The study shows that while prompt optimization can raise verifier pass rates, many accepted specifications remain behaviorally weak, either over‑ or under‑constraining inputs and outputs. Spec‑Harness also serves as a feedback mechanism that improves the quality of specifications generated by general‑purpose coding agents and a specialized JML agent.
arXiv AI
Oct 1
Self-Spec Verifiable Code Generation
arXiv:2609.39568v1 Announce Type: cross Abstract: Large language models (LLMs) may generate unreliable code on corner cases missed by testing, while formal verification can provide machine-checkable...
arXiv AI
Oct 9
Beyond Type-checking: Towards Holistic Evaluation of Formal Specification Generation
The paper presents a unified dataset of 350 Lean tasks to evaluate specification generation (SpecGen) for verifiable code, combining metrics of formal validity, reference similarity, equivalence, and behavioural adequacy. It highlights that restricting similarity measures to a subset of tasks can shift rankings, and shows that a perfect postcondition score can still miss critical input coverage, underscoring the need for separate feedback on input and output constraints.
arXiv AI
Jun 26
An Empirical Study of LLM-Generated Specifications for VeriFast
arXiv:2606. 26490v1 Announce Type: cross Abstract: Static verification tools can assure industrial scale software, but require significant human labor to write specifications.
arXiv AI
Sep 21
SWE-Proof: Can Language Models Resolve Real-World Issues with Machine-Checked Proofs?
arXiv:2609.21190v1 Announce Type: cross Abstract: Ensuring the correctness of LLM-generated code is a core challenge for modern software engineering. Benchmarks for agentic code generation check corr...
arXiv AI
Aug 18
ReLoop: Structured Modeling and Behavioral Verification for Reliable LLM-Based Optimization
arXiv:2602. 15983v3 Announce Type: replace-cross Abstract: Large language models (LLMs) can translate natural language into optimization code, but silent failures pose a critical risk: code that executes and returns solver-feasible solutions may encode semantically incorrect formulations---a feasibility--correctness gap reaching 90 percentage points on compositional problems.