arXiv AI
Jul 15
Tracing Agentic Failure from the Flow of Success
arXiv:2607. 12747v1 Announce Type: new Abstract: Failure attribution for LLM-based agentic systems, i.
When Evidence is Sparse: Weakly Supervised Early Failure Alerting in Dialogs and LLM-Agent Trajectories
arXiv:2606. 05414v1 Announce Type: cross Abstract: Early failure alerting requires deciding, while a dialog or agent trajectory is still unfolding, whether to flag it as likely to fail.
Related stories
arXiv AI
Jul 23
Stateful Guardrails for Multi-Turn LLM Systems: A Conversational Risk Accumulation Framework
arXiv:2607. 19361v1 Announce Type: cross Abstract: Most safety guardrails for large language models (LLMs) evaluate each prompt-response pair in isolation, which misses failures that arise only over a dialogue as benign turns compose into harm.
arXiv AI
Sep 15
Root-Cause Attribution Is a Search Problem: Continual Search for Long-Horizon Agent Failures
The paper introduces Continual Search, an iterative framework that guides large language models to persistently search for diagnostic evidence in long AI agent execution logs, addressing the limitations of one-shot judgments. Evaluated on four existing RCA benchmarks and a new large-scale dataset called MegaRCA-Mix, Continual Search consistently boosts attribution performance, achieving a 40% F1 improvement for GPT‑5.5 on MegaRCA‑Mix. The results show that effective search can outweigh raw model scale, enabling lower-tier models to outperform higher-tier ones in root‑cause attribution tasks.
arXiv Computation and Language
Sep 1
CAST: Critique-Aware Supervision for Training Reliable Long-Horizon Tool-Calling Agents
CAST is a critique‑aware training framework that transforms sparse task outcomes into action‑level supervision for both critique learning and policy optimization. By analyzing agent trajectories, CAST synthesizes structured rationales that explain action validity under partial observability, enabling the creation of richer training data. Fine‑tuned Qwen3‑family models trained with CAST show significant reliability gains, outperforming GPT‑OSS‑120B by over 10% on Retail tasks and improving Telehealth performance by 9% in an out‑of‑domain setting.
arXiv AI
Jun 3
StepFinder: A Temporal Semantic Framework for Failure Attribution in Multi-Agent Systems
arXiv:2606. 03467v1 Announce Type: new Abstract: LLM-based multi-agent systems exhibit remarkable collaborative capabilities in complex multi-step tasks.
arXiv Machine Learning
Jul 30
Forecasting Trajectory-Level Safety Risks in Black-Box Multi-Turn Interactions
arXiv:2607. 26820v1 Announce Type: new Abstract: As large language models (LLMs) evolve from standalone assistants into autonomous agents, ensuring their safety requires shifting beyond pointwise risk assessment to understand how risks emerge and unfold over long-horizon trajectories.
arXiv Machine Learning
Jun 10
FailureScope: Cross-Regime Behavioral Diagnosis of Language Model Weaknesses
arXiv:2606. 09878v1 Announce Type: new Abstract: Standard benchmarks report aggregate accuracy, but practitioners need to know which specific capabilities a model lacks.
arXiv AI
Sep 24
Beyond Unsafe Detection: Counterfactually Anchored Evidence Attribution for Multi-Turn LLM Safety Failures
The paper introduces a counterfactually anchored evidence attribution approach for multi‑turn large language model safety failures. It presents a new dataset of 1,762 conversations, including adversarial, benign twins, and high‑risk vocabulary variants, and trains a lightweight hierarchical model that accurately predicts safety violations and attributes them to specific user turns and token spans. The model achieves high detection performance (F1 = 0.988) and significantly reduces adversarial confidence when top‑attributed tokens are removed, while maintaining low false‑positive rates on benign conversations.
arXiv AI
Jun 2
FALAT: Tracing Failures in LLM Agent Trajectories via Dependency-Guided Search
arXiv:2606. 00765v1 Announce Type: new Abstract: LLM-based agents increasingly solve complex tasks through long trajectories involving reasoning steps, tool calls, and inter-agent communication.
arXiv AI
Sep 3
Monitoring Web Agents Without Internal Signals: Observable Trajectories and Key-Step Supervision
The paper introduces a method for monitoring web agents without relying on internal signals such as token logits. It proposes two observable trajectory representations—Macro features that capture cross‑step agent–environment interactions, and Micro features that assess consistency of intention, action, and expected state change via repeated black‑box queries. By labeling the first uncorrected critical error that leads to final failure as a key‑step boundary, the approach preserves valid early prefixes of failed trajectories and achieves risk prediction performance competitive with internal‑signal baselines across WebArena‑Lite and Online Mind2Web benchmarks.
arXiv AI
2d ago
DeFA: Dependency-Guided Failure Attribution for LLM Agents
DeFA is a dependency-guided framework that attributes failures in large language model agents by constructing an event dependency graph and a failure propagation graph from protocol relations and semantic dependencies. It identifies violating events, traces their sources and effects, and determines the decisive error, responsible agent, and error category. The method supports long trajectories through segmentation and has shown superior accuracy on text, image, and video tasks, while its diagnostic feedback can improve agent performance on subsequent tasks.
arXiv AI
Sep 1
AgentRx: Diagnosing AI Agent Failures from Execution Trajectories
arXiv:2602.02475v2 Announce Type: replace Abstract: AI agents often fail in ways that are difficult to localize because executions are probabilistic, long-horizon, multi-agent, and mediated by noisy...