arXiv AI By Aditya Shukla, Kartik Gupta, Ali Alavi, Pashootan Vaezipoor

Spec-Driven PBT: Deriving Semantic Invariants from Code Specifications for Agentic Property-based Testing

Read the original on arXiv AI →

Spec-Driven PBT is a testing approach that derives invariant properties from a natural‑language specification before inspecting the implementation to generate concrete tests. In a study of 25 ViBench applications, it uncovered a unique set of bugs that Code‑driven PBT missed, using 2.4× fewer property tests and finding roughly the same total number of bugs with little overlap between the two methods. The approach highlights how specifications can reveal bugs related to documented product workflows, complementing implementation‑focused testing.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI
Oct 31

Skeleton Coding: Program Structure as a Specification for Agents

The paper introduces Skeleton Coding, a method where developers jointly specify a program’s structure—its data types, modules, and functions—alongside behavioral tests. This approach allows the agent to first commit to a structure, narrowing the search space of possible programs, and then use tests to select the correct implementation. Empirical results across multiple models and benchmarks show that skeletons can dramatically reduce the number of required tests, improve accuracy, and lower implementation costs.

By Aditya Golatkar, Alessandro Achille, Renos Zabounidis, Vlad Sobal, Yangruibo Ding, Baishakhi Ray, Jonathan Buck, Guy Van den Broeck, Maryam Fazel, Pietro Perona, Wei Xia, Stefano Soatto
Hugging Face Trending Papers
Jul 29

SpecFirst: Behavioral Specification Elicitation as a First-Class Step in Agent-Based Program Synthesis from Scratch

LLM-based agents excel at software engineering tasks where an existing codebase provides context, but constructing a program from scratch remains fundamentally harder. Recent benchmarks such as ProgramBench quantify this gap: given only natural-language documentation and an execute-only binary as a behavioral oracle, even frontier models solve fewer than 1% of instances.

arXiv AI
Oct 28

Benchmarking LLM Agents for Behaviorally Adequate Test Generation from Software Requirements

The paper introduces Req2Test-Bench, a benchmark of 5,086 instances linking natural‑language Gherkin requirements to developer‑written test implementations across five languages and four BDD frameworks. It evaluates eight agent–model–reasoning setups, finding that while the best configuration (Codex with DeepSeek‑V4.1‑Flash) produces passing tests for 82.8% of cases, only 33.8% are behaviorally adequate, as measured by kill‑completeness. The study highlights diminishing returns from scaling agents, the impact of under‑specified requirements, and the potential of requirement‑grounded harnesses to improve test generation.

By Binhang Qi, Yiming Liu, Yun Lin, Yangjie Zhou, Hailong Sun, Jin Song Dong