AI agents can make incorrect decisions or waste resources even when their tools and infrastructure work normally. Their nondeterministic behavior makes these problems difficult to reproduce and diagno...
arXiv:2602.02475v2 Announce Type: replace
Abstract: AI agents often fail in ways that are difficult to localize because executions are probabilistic, long-horizon, multi-agent, and mediated by noisy...
By Shraddha Barke, Arnav Goyal, Alind Khare, Avaljot Singh, Suman Nath, Chetan Bansal
AgentEval is a system that tracks a contract-bound assessment record throughout a stateful agent’s tool execution, capturing state, events, and evidence to determine whether obligations are satisfied, violated, or unresolved. In 32 tool‑world administrations, independent checks confirmed all 30 complete episodes, while controlled interventions on 48 evidence views uncovered 36 binding faults that hash‑only checks missed. A public‑benchmark study on 192 held‑out executions identified ten completed subgoals within broader task failures and highlighted the impact of missing evidence on overall task success.
By Ruiqi Gao (Alibaba Cloud, Alibaba Group)
arXiv:2610.16725v1 Announce Type: new
Abstract: Improving an LLM agent requires knowing what to change when it fails. Similar failures can require different fixes, and resolving one problem can expos...
By Wenfeng Wang, Jianrui Shi, Mingang Guo, Jiayuan Ma, Yuqi Lu, Jiajun Sun, Guojiang Zhao, Jing Li
arXiv:2609.06063v1 Announce Type: new
Abstract: AI Agents are increasingly deployed in real-world settings, where they interact with external tools and make sequential decisions with limited human ov...
By Vittoria Vineis, Fabiano Veglianti, Lorenzo Antonelli, Claudia Di Carlo, Matteo Silvestri, Gabriele Tolomei
The paper introduces NetArtifactBench, a benchmark designed to evaluate whether AI agents can detect and repair inconsistencies in network experiment records while preserving supported claims. It tests 23 agent configurations on 52 instances with injected inconsistencies, finding an average pass rate of 65.3 % but no runtime exceeding 30 % for complex repairs that require recovering implicit relations and propagating changes across artifacts. The results highlight a clear distinction between local corrections and full record-level repair, leading the authors to argue that artifact integrity should be a primary design and evaluation criterion for AI agents in network systems.
By Tianzhu Zhang, Weichen Tao, Changgang Zheng, Yusheng Zheng, Long Chen, Xiaoyi Fan, Meikang Qiu