arXiv AI
Eliciting Specifications: When Does Interactive Clarification Help?
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Jul 1
Beyond Compilation: Evaluating Faithful Natural-Language-to-Lean Statement Formalization
arXiv:2606. 31002v1 Announce Type: new Abstract: Theorem-proving benchmarks evaluate proof search against fixed formal statements, but natural-language-to-Lean formalization must generate the formal statement itself.
arXiv AI
Oct 9
Beyond Type-checking: Towards Holistic Evaluation of Formal Specification Generation
The paper presents a unified dataset of 350 Lean tasks to evaluate specification generation (SpecGen) for verifiable code, combining metrics of formal validity, reference similarity, equivalence, and behavioural adequacy. It highlights that restricting similarity measures to a subset of tasks can shift rankings, and shows that a perfect postcondition score can still miss critical input coverage, underscoring the need for separate feedback on input and output constraints.
arXiv AI
Oct 5
ClarifyCodeBench: Evaluating LLMs on Clarifying Ambiguous Requirements for Code Generation
arXiv:2607.00711v2 Announce Type: cross Abstract: Large Language Models have emerged as programming assistants. However, the efficacy of code generation is constrained by the quality of input require...
arXiv AI
5d ago
AM-Bench: Understanding Agents under Task Underspecification across Models and Harnesses
arXiv:2610.18090v1 Announce Type: new Abstract: Ambiguous or incomplete task specifications complicate both execution and clarification for language-model agents. Yet agents' responses to these infor...
arXiv AI
Sep 11
Spec-Harness: Measuring and Improving Behavioral Adequacy of LLM-Synthesized Formal Specifications
Spec‑Harness evaluates how well large language models (LLMs) synthesize Java Modeling Language (JML) specifications by measuring behavioral adequacy across precondition and postcondition correctness and completeness. The study shows that while prompt optimization can raise verifier pass rates, many accepted specifications remain behaviorally weak, either over‑ or under‑constraining inputs and outputs. Spec‑Harness also serves as a feedback mechanism that improves the quality of specifications generated by general‑purpose coding agents and a specialized JML agent.
arXiv AI
Jun 8
Lean4Agent: Formal Modeling and Verification for Agent Workflow and Trajectory
arXiv:2606. 06523v1 Announce Type: new Abstract: Equipping Large Language Models (LLMs) to execute reliable multi-step workflows has become a central challenge in artificial intelligence.