The paper introduces rebuild‑dossier, an open‑source tool that locks an application’s real interface before code is written and enforces one‑test‑at‑a‑time building through automated checks. In experiments, a compliant agent failed a held‑back test while a rule‑breaking agent passed, showing that a passing test suite can be gamed. The study also demonstrates that the automated check mechanism, rather than interface‑locking alone, is crucial for reliable rebuilds, and that multi‑level verification catches errors that single‑level checks miss.
By Parker Fawcett
arXiv:2609.22664v1 Announce Type: cross
Abstract: Research on large language model agents for penetration testing is evaluated almost entirely by capability: whether the agent captures a flag or repr...
By Joas Antonio dos Santos Barbosa
arXiv:2609.21190v1 Announce Type: cross
Abstract: Ensuring the correctness of LLM-generated code is a core challenge for modern software engineering. Benchmarks for agentic code generation check corr...
By George Ma, Benjamin Mikek, Haoyu Li, Ferhat Erata, Yuhao Zhang, Zeren Shui, Behrooz Omidvar Tehrani, Jun Huan, Murali Krishna Ramanathan, Somayeh Sojoudi, Hao Zhou, Anoop Deoras
The paper presents a method for turning expert diagnoses of verification failures into reusable guidance for coding agents. By combining executable language definitions in the K framework with a set of procedures for constructing specifications, repairing proofs, and auditing their adequacy, the authors achieve a 164/164 success rate on the HumanEval benchmark after two targeted repairs. They further demonstrate that audits can detect defects missed by successful proofs and evaluate the approach on KleverBench and Optimism proofs, highlighting both progress and remaining challenges.
By Yuqing Zhai, Xiaohong Chen, Lingming Zhang, Sriram Vishwanath, Grigore Rosu
Consort is a spec‑first, test‑driven agent framework that enforces engineering discipline through immutable controls, a deterministic orchestrator, and human‑approved gates. It operates on live database branches, guiding role agents through a spec‑first design lane and a test‑driven build lane. The framework claims that enforcing tests and gates in code keeps agent‑written code honest and verifiable, while specialized roles make it maintainable.
By Kevin Hartman
The paper introduces Generative Test-Driven Development (GTDD), a method where a separate testing agent continuously generates new inputs after each coding agent’s implementation, guided by a human-specified behavioral contract and prior feedback. A trusted evaluator validates these inputs, provides reduced counterexamples, and stores them for regression testing, ensuring that development repeatedly confronts failures beyond the initial examples. Experimental results on a stateful key-value store show that policies regenerating tests during development achieve lower mean failure rates than a single-generation policy, while providing the tester with the candidate’s source code does not yield additional improvement.
By Masahiro Kato