The paper investigates how AI agents that generate smart contracts from plain‑English descriptions can lose intended behavior, a critical issue because smart contracts are immutable once deployed. The authors design an execution‑based evaluation that replays transactions on both the AI‑written contract and the original, measuring agreement and identifying discrepancies. They find that common grading tools miss many mistakes, and propose an elicitation‑first approach where the agent asks a short checklist of questions before coding, which significantly improves correctness across multiple AI models.
arXiv:2606. 13468v1 Announce Type: cross Abstract: AI coding agents are increasingly used to generate pull requests (PRs) that propose code fixes in software projects.
The paper introduces Verifiable Agent Insurance (VAI), a framework that lets AI agent providers commit to policy compliance through smart contracts and escrow funds on a blockchain. When a user detects a policy violation, a verifier reviews execution traces logged by the agent harness; if the violation is confirmed, the escrow is paid to the user. Experiments on three agent benchmarks show VAI detects 42.2% of violations, rising to 83.0% when agents emit explicit structured records.
The paper introduces CONSTRAINTDIAL, a framework that automates the lifelong construction of smart contract security tasks through protocol mutation. It reconstructs contracts locally and uses attack execution to modify required calls or inputs, ensuring the original attack fails while maintaining exploitability and normal functionality. Applying this method to 203 public cases yielded 3,045 task inputs across detect, exploit, and patch categories, reducing reliance on copied answers by 10.9 percentage points and lowering the fraction of tasks solved by online copying by 9.3 points.
AgentSWE is a new benchmark for agent software engineering that evaluates coding agents across three lifecycle stages: creating an agent from a natural‑language requirement, editing existing agent codebases, and optimizing toward a measurable target. The benchmark consists of 25 commission‑style tasks, where builders iterate on a few illustrative cases and are then tested on held‑out cases that cover the full requirement, with hard pass/fail gates or objective‑based grading. Experiments show that current coding agents fail to meet the requirements, revealing common failure modes such as overfitting, incomplete runtimes, and unsupported completion claims.
The paper argues that answer accuracy alone is insufficient for evaluating large language model (LLM) data agents, especially in structured-data tasks where a correct answer can be produced by an invalid trace. It introduces Trace Integrity as a reliability criterion that ensures the computation behind an answer is explicit, executable, schema-valid, operator-faithful, replayable, answer-consistent, and auditable. The authors operationalize this concept with execution contracts and present the CAIT (Correct Answer / Invalid Trace) Rate to quantify how often answer-only evaluations mistakenly reward unsupported outputs, demonstrating that accuracy, trace validity, and silent-failure risk are distinct signals.
The paper investigates the reliability of software produced by agentic AI by comparing AI-generated versions of ten well-known Linux utilities to their human-written counterparts. Using fuzz testing (both black-box and coverage-guided AFL++), the authors find that AI-generated code is often as reliable or more reliable than the latest human versions, with fewer memory errors but a higher incidence of hangs. The study emphasizes that robust AI-generated software requires careful prompting, skilled human oversight, and that the AI workflow can serve as a cost-effective specification for sustainable code.
Answer accuracy is an insufficient reliability signal for LLM data agents. In structured-data tasks, a benchmark-correct answer can be produced by an invalid trace. This paper introduces Trace Integri...
The paper introduces rebuild‑dossier, an open‑source tool that locks an application’s real interface before code is written and enforces one‑test‑at‑a‑time building through automated checks. In experiments, a compliant agent failed a held‑back test while a rule‑breaking agent passed, showing that a passing test suite can be gamed. The study also demonstrates that the automated check mechanism, rather than interface‑locking alone, is crucial for reliable rebuilds, and that multi‑level verification catches errors that single‑level checks miss.
arXiv:2607. 16345v1 Announce Type: cross Abstract: Modern agentic systems increasingly rely on skills: installable packages of natural language and code that teach an LLM agent to perform a domain task.
arXiv:2610.14477v1 Announce Type: cross Abstract: Robustness tests for tool-using agents often change the agent's environment, for example by hiding a hostile note in an invoice, and then judge the a...
arXiv:2607. 13034v1 Announce Type: new Abstract: Large language model (LLM) agents increasingly automate multi-step engineering and informatics workflows, yet they rarely ask how much effort a task actually requires.
arXiv:2608.26036v2 Announce Type: replace Abstract: Large language model (LLM) data agents answer structured-data questions by generating SQL, invoking tools, executing queries, and summarizing resul...