arXiv:2606. 13468v1 Announce Type: cross Abstract: AI coding agents are increasingly used to generate pull requests (PRs) that propose code fixes in software projects.
By Mahmoud Abujadallah, Ali Arabat, Mohammed Sayagh
The paper introduces Verifiable Agent Insurance (VAI), a framework that lets AI agent providers commit to policy compliance through smart contracts and escrow funds on a blockchain. When a user detects a policy violation, a verifier reviews execution traces logged by the agent harness; if the violation is confirmed, the escrow is paid to the user. Experiments on three agent benchmarks show VAI detects 42.2% of violations, rising to 83.0% when agents emit explicit structured records.
By Cheng Hua, Taro Tsuchiya, Nicolas Christin, Giulia Fanti
The paper introduces CONSTRAINTDIAL, a framework that automates the lifelong construction of smart contract security tasks through protocol mutation. It reconstructs contracts locally and uses attack execution to modify required calls or inputs, ensuring the original attack fails while maintaining exploitability and normal functionality. Applying this method to 203 public cases yielded 3,045 task inputs across detect, exploit, and patch categories, reducing reliance on copied answers by 10.9 percentage points and lowering the fraction of tasks solved by online copying by 9.3 points.
By Zifan Peng, Peng Huang, Jingyi Zheng, Yingjie Xue, Xuechao Wang, Xinlei He, Xia Hu, Dongrui Liu
AgentSWE is a new benchmark for agent software engineering that evaluates coding agents across three lifecycle stages: creating an agent from a natural‑language requirement, editing existing agent codebases, and optimizing toward a measurable target. The benchmark consists of 25 commission‑style tasks, where builders iterate on a few illustrative cases and are then tested on held‑out cases that cover the full requirement, with hard pass/fail gates or objective‑based grading. Experiments show that current coding agents fail to meet the requirements, revealing common failure modes such as overfitting, incomplete runtimes, and unsupported completion claims.
By Jiahao Wang, Hongjin Qian, Yuyang Hu, Jiajun Zhang, Zheng Liu
The paper argues that answer accuracy alone is insufficient for evaluating large language model (LLM) data agents, especially in structured-data tasks where a correct answer can be produced by an invalid trace. It introduces Trace Integrity as a reliability criterion that ensures the computation behind an answer is explicit, executable, schema-valid, operator-faithful, replayable, answer-consistent, and auditable. The authors operationalize this concept with execution contracts and present the CAIT (Correct Answer / Invalid Trace) Rate to quantify how often answer-only evaluations mistakenly reward unsupported outputs, demonstrating that accuracy, trace validity, and silent-failure risk are distinct signals.
By Srimonti Dutta, Akshata Kishore Moharir
The paper investigates the reliability of software produced by agentic AI by comparing AI-generated versions of ten well-known Linux utilities to their human-written counterparts. Using fuzz testing (both black-box and coverage-guided AFL++), the authors find that AI-generated code is often as reliable or more reliable than the latest human versions, with fewer memory errors but a higher incidence of hangs. The study emphasizes that robust AI-generated software requires careful prompting, skilled human oversight, and that the AI workflow can serve as a cost-effective specification for sustainable code.
By Ayesha Shafique, Barton P. MIller, Elisa R. Heymann