arXiv AI
Sep 3

ClaimReceipt: Verifying Evidence Sufficiency and Coverage in Agent Evaluations

The paper introduces ClaimReceipt, a specification and verifier that checks whether a claim in an agent evaluation can be recomputed from retained evidence (sufficiency) and whether the evidence covers the entire experiment set (coverage). Using the CR‑2 verifier on 1,392 historical records, the authors demonstrate accurate reproduction of audit verdicts, non‑redundant field groups, and zero false positives on semantic faults. In a prospective CR‑3 run, the system correctly flags missing receipts and preserves coverage when private evidence is withheld, while adding minimal overhead to inference time and transaction size.

By Peiying Zhu, Sidi Chang
arXiv AI
Aug 24

Calibrating Criterion Revision in LLM Agents: Failure Modes and a Trace-Anchored Protocol

The paper introduces a framework for evaluating how large language model agents revise their success criteria after failures, defining five non‑compensatory conditions that must be met for a criterion revision to be considered valid. Using the CMB‑0.1 protocol, the authors test twelve cross‑domain scenarios across four system configurations, finding that no model trial satisfies all five conditions and highlighting specific failure modes such as zero‑state reconstruction and inadequate intervention sensitivity. They propose a more stringent trace‑anchored CMB‑0.4 protocol to better isolate and measure criterion revision in future studies.

By Guodong Xu
arXiv AI
Aug 19

PACE: Policy-Attested Contract Execution for Safe AI Agents in Decentralized Finance

The paper introduces PACE (Policy‑Attested Contract Execution), a framework that sits between large‑language‑model (LLM) based autonomous AI agents and on‑chain DeFi operations. PACE defines typed transaction intents, a deterministic policy verifier, and signed Policy Decision Records (PDRs) that cryptographically bind an approved intent, policy, and simulation report to the exact on‑chain execution bytes, providing replay and expiration protection. In evaluations across 40 tasks and six baselines, PACE achieves zero unsafe executions and zero false positives, outperforming unguarded agents by a large margin.

By Rabimba Karanjai (Larry), Yang Lu (Larry), Richard Williamson (Larry), Hemanth Hm (Larry), Prakhar Mehrotra (Larry), Lei Xu (Larry), Weidong (Larry), Shi
arXiv AI
1d ago

Who Verifies the Graph? Misspecification Attacks on Causal Action Verification for Language Agents

The paper investigates how causal action verifiers, which guard language agents’ tool calls by checking identifiability against a committed action‑state graph, can be compromised through small graph misspecifications. By removing a single bidirected edge or reversing an arrowhead, the authors demonstrate that a verifier (CIVeX) that originally had zero false executions can suffer false execution rates up to 48.9%, with most of those executions being harmful and overall utility dropping dramatically. An additional attestation step that samples executions can detect these attacks with few false alarms, but it also leads to many wrongful rejections that reduce beneficial actions and incur significant experimental costs. whyItMatters":"The study shows that even minor errors in the verifier’s underlying graph can drastically undermine safety and performance, highlighting the need for robust auditing mechanisms."

By Fabio Rovai
arXiv AI
Sep 25

Where Does Exactly-Once Live? Model, Harness, and Tool-Contract Effects on Duplicate Side Effects in LLM Agents

The paper investigates where exactly‑once semantics should be enforced for tool‑using agents—within the model, the agent harness, or the tool contract—by evaluating 25,930 episodes across nine models, three harnesses, two contract variants, and fifteen recovery conditions. Using the LIMBO sandbox, the study shows that when an immediate read‑back is available, frontier models rarely duplicate lost acknowledgements, whereas weaker models do; when read‑back is unavailable, the contract’s idempotency keys explain most duplicate behavior. The authors prove that verification‑only policies cannot guarantee exactly‑once under late commits without bounded in‑flight time, and that waiting only helps when delays are short and predictable. whyItMatters":"The findings clarify that enforcing exactly‑once semantics largely depends on the tool contract and fault type, guiding designers on where to focus reliability mechanisms for LLM agents."

By Jiapeng Li