AgentEval is a system that tracks a contract-bound assessment record throughout a stateful agent’s tool execution, capturing state, events, and evidence to determine whether obligations are satisfied, violated, or unresolved. In 32 tool‑world administrations, independent checks confirmed all 30 complete episodes, while controlled interventions on 48 evidence views uncovered 36 binding faults that hash‑only checks missed. A public‑benchmark study on 192 held‑out executions identified ten completed subgoals within broader task failures and highlighted the impact of missing evidence on overall task success.
arXiv:2610.07753v1 Announce Type: cross Abstract: Tool-using agents make consequential changes to external state, yet correct outcomes do not guarantee that their actions were supported by evidence e...
Shared memory coordinates agents' actions, but correct records do not establish that those actions satisfy task requirements. Memory governance and failure diagnosis regulate or inspect recorded infor...
arXiv:2610.08101v1 Announce Type: new Abstract: Shared memory coordinates agents' actions, but correct records do not establish that those actions satisfy task requirements. Memory governance and fai...
arXiv:2606. 04990v1 Announce Type: cross Abstract: Large language model (LLM)-based agents increasingly solve complex tasks by interacting with external tools, retrieval systems, memory modules, environments, and other agents.
arXiv:2606. 02965v2 Announce Type: replace Abstract: As large language models gain tool access and are deployed as autonomous agents capable of editing records, executing transactions, and modifying infrastructure, we still evaluate them based on the sole metric of task completion.
arXiv:2608. 13867v1 Announce Type: cross Abstract: AI coding agents are commonly evaluated as models but deployed as systems.
LEDGER is a tracing and review system for large language model agents that constructs layered trace graphs from observed sessions. It groups raw trace records into Evidence Nodes and Workflow Nodes, anchors artifacts as evidence, and adds typed semantic edges linking claims to supporting actions, artifacts, and checks. The resulting traces reveal workflow decisions, artifact lineage, repair steps, validation coverage, and claim‑support paths for evidence‑centered audit.
arXiv:2606. 04990v2 Announce Type: replace-cross Abstract: Large language model (LLM)-based agents are evolving from passive text generators into autonomous systems capable of planning, tool use, retrieval, memory access, environmental interaction, and multi-agent collaboration.
arXiv:2609.13672v1 Announce Type: new Abstract: AI agents can be interrupted while editing files, calling tools, or carrying out multi-step tasks. Restarting repeats completed work, but continuing fr...
Diviner is an agentic root‑cause analysis system designed for large‑scale distributed infrastructures. It organizes incident investigation into a four‑stage reliability lifecycle, leveraging a corpus of reviewed incidents and service‑owner expertise to create scoped, versioned skills, workflows, typed tools, and an incident knowledge graph. The system employs adaptive lead and context‑isolated workers to apply relevant knowledge to real‑time evidence, evaluates investigation trajectories against responder‑reviewed outcomes, and updates operational knowledge through feedback loops that require replay, quality checks, and owner approval.
The paper introduces rebuild‑dossier, an open‑source tool that locks an application’s real interface before code is written and enforces one‑test‑at‑a‑time building through automated checks. In experiments, a compliant agent failed a held‑back test while a rule‑breaking agent passed, showing that a passing test suite can be gamed. The study also demonstrates that the automated check mechanism, rather than interface‑locking alone, is crucial for reliable rebuilds, and that multi‑level verification catches errors that single‑level checks miss.
The paper "When Agents Act Unwatched: The Reduced‑Supervision Paradox in Agentic AI" discusses how the promise that AI systems will continue acting after users stop watching creates an accountability inversion. It argues that as stepwise supervision recedes, verification shifts into the runtime infrastructure—authority, records, interrupts, outcome checks, and repair—forming what the authors call the reduced‑supervision paradox. A 63‑artifact audit across research papers and engineering sources shows that agents’ action surfaces are more visible than the mechanisms needed to hold them accountable, with tool mediation and monitoring traces appearing in 40 and 37 artifacts, while checkpoint placement, validator independence, recovery, and contestability are rarely visible. "whyItMatters":"The study highlights that observable action paths can replace accountability when verification is moved onto users after meaningful intervention is no longer possible."