The paper introduces Evidence-Carrying Termination (ECT), a method that allows tool‑using large language model agents to declare a task complete only when a typed certificate links every required answer claim to valid, in‑scope trace evidence and a deterministic replay confirms the claimed value. In controlled experiments across 48 synthetic tasks and 576 trajectories, ECT eliminated unsafe completions and premature unsupported terminations, outperforming existing termination critics and controllers while maintaining comparable supported completion rates.
arXiv:2608. 03222v1 Announce Type: cross Abstract: Software engineering (SWE) agents resolve repository-level issues through long trajectories that grow increasingly expensive as context accumulates.
arXiv:2610.17633v1 Announce Type: new Abstract: Work on debugging LLM agents treats the "first mistake" marked by a human or LLM judge as ground truth for where a trajectory failed. We ask whether su...
Software engineering (SWE) agents resolve repository-level issues through long trajectories that grow increasingly expensive as context accumulates. Failed runs tend to be longer and exhibit redundant exploration or looping, suggesting that some failures may be detectable before completion.
The paper investigates whether human or LLM‑judge‑identified ‘first mistake’ points truly indicate failure causes in LLM agents. By reconstructing intermediate states and computing a prefix value V(h_t) for each step, the authors identify a certified point of no return (PNR) and compare it to judge‑labeled steps. Their experiments on 150 SWE‑bench tasks show that judge labels often miss the decisive step, while restarting at the measured PNR improves success rates from 0.44 to 0.57.
The paper introduces NAQD‑Env, a synthetic benchmark designed to test language agents’ ability to selectively withdraw and resume actions when new evidence, permissions, or stop instructions arise. It evaluates models against a deterministic reference policy across eleven dependency families, measuring policy agreement, task value, withdrawal, resumption, and event reporting. Experiments on 350 scenarios show low withdrawal recall, no valid resumption, and limited policy alignment, highlighting the need to treat selective withdrawal as a distinct reliability component.
arXiv:2607. 07405v1 Announce Type: new Abstract: Tool-using LLM agents can violate the very policies they are deployed to enforce while appearing to complete the task successfully.
arXiv:2609.38266v1 Announce Type: cross Abstract: Agentic large language models (LLMs) now move money through tools, yet the record of what they did is usually a trace their own process emits beside...
The paper introduces ClaimReceipt, a specification and verifier that checks whether a claim in an agent evaluation can be recomputed from retained evidence (sufficiency) and whether the evidence covers the entire experiment set (coverage). Using the CR‑2 verifier on 1,392 historical records, the authors demonstrate accurate reproduction of audit verdicts, non‑redundant field groups, and zero false positives on semantic faults. In a prospective CR‑3 run, the system correctly flags missing receipts and preserves coverage when private evidence is withheld, while adding minimal overhead to inference time and transaction size.
The paper introduces a framework for evaluating how large language model agents revise their success criteria after failures, defining five non‑compensatory conditions that must be met for a criterion revision to be considered valid. Using the CMB‑0.1 protocol, the authors test twelve cross‑domain scenarios across four system configurations, finding that no model trial satisfies all five conditions and highlighting specific failure modes such as zero‑state reconstruction and inadequate intervention sensitivity. They propose a more stringent trace‑anchored CMB‑0.4 protocol to better isolate and measure criterion revision in future studies.
arXiv:2606. 09863v1 Announce Type: new Abstract: LLM agents can fail silently by asserting task completion when the environment state shows otherwise.
Praxa is an evidence‑bound harness for governed AI agent execution that explicitly represents states such as proposal, authority, dispatch, verified external effect, and promotion through deterministic admission, brokered execution, external read‑back, reconciliation, and reviewed promotion. The authors report four evidence lanes: a repository‑local audit passing all unit and Workerd tests; a pilot on 12 curated tasks where both baseline and reliability‑layer arms passed 17 of 36 trials; a coordination‑proxy comparison where both baseline and a source‑authored candidate completed all 180 trials with equal accuracy but the candidate used fewer tokens and steps; and deployed source/configuration evidence showing bounded reflection, recall accounting, memory compilation, and tool‑health paths. None of the evidence demonstrates superiority in security, safety, or user benefit. whyItMatters:"Praxa provides a testable architecture that makes authority‑to‑effect transitions explicit, offering a framework for verifying AI agent behavior, though current evidence does not prove improved security or performance."
The paper investigates the safety of Code World Models, where a language model generates executable world models that a planner uses. It shows that accepting a model based on sampled transitions only guarantees sample consistency, not full safety, because the probability of missing critical events decays as (1‑r)^N. Experiments on hybrid instruments reveal that omitted mode‑boundaries can severely limit planner performance, and that even sophisticated LLMs (GPT‑5.x) struggle to repair such omissions in higher‑dimensional settings.