The paper "Signals: Continuous Diagnostics of AI Agents" introduces a method for diagnosing problems in AI agents by analyzing agentic traces. It defines 45 problem types across eight families, categorizes findings into engineering issues called Signals, and assesses their severity. The method was evaluated on public agent executions, achieving a 91.43% F1 score in matching Signals to reference issues, outperforming a one-pass LLM control by about 18 percentage points.
By Amey Rambatla, Vatsal Goel, Sagar Gaur, Siddhant Loya, Anurag Mishra, Asif Mohammad
arXiv:2609.06063v1 Announce Type: new
Abstract: AI Agents are increasingly deployed in real-world settings, where they interact with external tools and make sequential decisions with limited human ov...
By Vittoria Vineis, Fabiano Veglianti, Lorenzo Antonelli, Claudia Di Carlo, Matteo Silvestri, Gabriele Tolomei
arXiv:2606. 02282v1 Announce Type: new Abstract: Orchestrating Large Language Models into Multi-Agent Systems (LLM-MAS) has unlocked remarkable reasoning capabilities, yet emergent failures and hallucinations that resist characterisation block their deployment in safety-critical domains -- a gap made legally untenable by emerging AI regulation.
By I\~naki Dellibarda Varela, R. Sendra-Arranz, Pablo Romero-Sorozabal, J. M. Valverde-Garc\'ia, Annemarie F. Laudanski, \'Alvaro Guti\'errez, Eduardo Rocon, Manuel Cebrian
arXiv:2609.06835v1 Announce Type: cross
Abstract: Agentic AI systems execute complex tasks through long-horizon workflows of planning, tool use, and multi-agent coordination. Task failures in these s...
By Chaoyu Zhang, Hexuan Yu, Heng Jin, Shanghao Shi, Ning Zhang, Yi Shi, Yulia R. Gel, Y. Thomas Hou, Wenjing Lou
arXiv:2602.02475v2 Announce Type: replace
Abstract: AI agents often fail in ways that are difficult to localize because executions are probabilistic, long-horizon, multi-agent, and mediated by noisy...
By Shraddha Barke, Arnav Goyal, Alind Khare, Avaljot Singh, Suman Nath, Chetan Bansal
arXiv:2606. 07054v1 Announce Type: cross Abstract: Autonomous LLM agents can pursue hidden malicious objectives through sequences of individually benign actions, making sabotage difficult to detect using standard trajectory-level monitoring.
By Vijitha Mittapalli, Shreyaa Jayant Dani, Satya Srujana Pilli, Snigdha Ansu, Mohammadreza Teymoorianfard, Franck Dernoncourt, Hongjie Chen, Yu Wang, Ryan A. Rossi, Nesreen K. Ahmed
The paper introduces TraceTrial-Bench, a benchmark that evaluates the reliability of AI agent harnesses across seven components—context, execution, governance, lifecycle, observability, tools, and verification—using 95 expert-level tasks with executable or state-based ground truth. It employs a Trajectory Tracing module to record agent execution, a two-layer evaluation framework to separately assess agent performance and harness reliability, and a Multi-source Evidence Acquisition module to gather evidence for LLM-based evaluation. The benchmark’s analysis of 3,420 runs across four harnesses and nine models shows harness scores between 59.8% and 72.0%, with 31.1% of runs succeeding despite failing at least one harness criterion, highlighting the need for auditable and reproducible harness evaluation.
By Zhiyong Cao, Hengjie Zhu, Yongxin Wang, Xuchen Li, Jintao Li, Shuxia Liu, Dunqiang Liu, Bo Wang, Kairan Zhang, Cong Lin, JiangDi Yu, Siyuan Liu, Sijia Yao, Yi-Liang Wei, Lingxu Ma, Aijia Wang, Anxin Chen, Zixian Wei, Xiang Ruan, Ruizhi Xu, Ruqian Shi, Baihua Xiao, Sicheng Zhou
arXiv:2608. 16411v1 Announce Type: cross Abstract: LLM-based agents are rapidly moving from research prototypes into the core business processes of organizations, but these agents pose deployment risks to security, compliance, and functionality.
By Yintong Huo, Rangeet Pan, Abhik Roychoudhury
arXiv:2606. 29026v1 Announce Type: new Abstract: Multi-agent AI systems can improve answer selection by allowing different language models to exchange reasoning traces, revise initial predictions, and support a final decision.
By Shahnewaz Karim Sakib, Anindya Bijoy Das
arXiv:2610.16725v1 Announce Type: new
Abstract: Improving an LLM agent requires knowing what to change when it fails. Similar failures can require different fixes, and resolving one problem can expos...
By Wenfeng Wang, Jianrui Shi, Mingang Guo, Jiayuan Ma, Yuqi Lu, Jiajun Sun, Guojiang Zhao, Jing Li
The article reports a case study of a production healthcare platform that was built using software‑engineering agents and overseen by an operator lacking formal software training. Over time the system evolved into a meta‑agent workflow where one agent wrote code, other agents supervised and reviewed it, and project rules carried lessons forward. The study found that tests, monitors, and reviewing agents were fallible—some measured proxies instead of outcomes, audits failed silently, checks disappeared from reports, and an automated repair caused operational disruption—highlighting that human control relied on aligning the intended outcome, the evidence used to judge it, the agents’ permissions, and the final human decision around the same objective.
By Lindsey Ferris, Sierra Bonilla
arXiv:2605.29442v2 Announce Type: replace-cross
Abstract: AI coding agents increasingly act directly within software environments, yet existing analyses of their failures rely on benchmark trajectori...
By Ningzhi Tang, Chaoran Chen, Gelei Xu, Yiyu Shi, Yu Huang, Collin McMillan, Tao Dong, Toby Jia-Jun Li