arXiv AI
Jul 15
Tracing Agentic Failure from the Flow of Success
arXiv:2607. 12747v1 Announce Type: new Abstract: Failure attribution for LLM-based agentic systems, i.
Can Failed Trajectories Reveal Where Agents Go Wrong? Agentic Failure Attribution Without Step-Level Labels
The paper introduces SOAP, a method for attributing failures in large language model agents without requiring step-level labels or a separate set of successful trajectories. SOAP builds a spectral reference space from LLM representations to assign base error scores to each step, then uses attention-based soft dependencies to propagate downstream error signals back to their likely upstream causes. Experiments demonstrate that SOAP outperforms existing baselines in accurately identifying decisive steps responsible for task failures.
Related stories
arXiv AI
Sep 1
Detect Before You Attribute: Cascade Failure Attribution for Multi-Agent Systems
arXiv:2608.29646v1 Announce Type: new Abstract: Large language model (LLM)-based agents have shown strong potential in solving complex tasks through multi-step reasoning, yet they remain vulnerable t...
arXiv AI
Jun 2
FALAT: Tracing Failures in LLM Agent Trajectories via Dependency-Guided Search
arXiv:2606. 00765v1 Announce Type: new Abstract: LLM-based agents increasingly solve complex tasks through long trajectories involving reasoning steps, tool calls, and inter-agent communication.
arXiv AI
Jun 3
StepFinder: A Temporal Semantic Framework for Failure Attribution in Multi-Agent Systems
arXiv:2606. 03467v1 Announce Type: new Abstract: LLM-based multi-agent systems exhibit remarkable collaborative capabilities in complex multi-step tasks.
arXiv AI
Sep 7
DCFA: Dual-view Causal-inspired Attribution for Failure Reasoning in LLM-based Multi-agent Systems
The paper introduces DCFA, a training‑free framework for attributing failures in large language model‑based multi‑agent systems. DCFA uses a global module to build causal‑inspired dependency graphs from system traces, pinpointing the earliest decisive error, and a local module that refines this attribution through counterfactual reasoning. Experiments on the Who&When benchmark across six LLMs demonstrate that DCFA improves step‑level accuracy by up to 8.27% over existing baselines.
arXiv AI
Oct 2
DeFA: Dependency-Guided Failure Attribution for LLM Agents
DeFA is a dependency-guided framework that attributes failures in large language model agents by constructing an event dependency graph and a failure propagation graph from protocol relations and semantic dependencies. It identifies violating events, traces their sources and effects, and determines the decisive error, responsible agent, and error category. The method supports long trajectories through segmentation and has shown superior accuracy on text, image, and video tasks, while its diagnostic feedback can improve agent performance on subsequent tasks.
arXiv AI
Oct 9
ReCast: Attribution-Oriented Step Representation Learning for LLM-Based Agent Systems
ReCast is a new method for learning step representations in large language model (LLM) based agent systems, aimed at improving failure attribution by transforming hidden states into attribution‑oriented representations. The approach selects relevant layers, builds pattern and deviation features, and trains an encoder with contrastive and ranking objectives. In experiments on four benchmarks, ReCast outperforms the strongest baseline, achieving higher Hit@1 scores on the Who&When Algorithm and Handcrafted datasets.
arXiv AI
Sep 1
AgentRx: Diagnosing AI Agent Failures from Execution Trajectories
arXiv:2602.02475v2 Announce Type: replace Abstract: AI agents often fail in ways that are difficult to localize because executions are probabilistic, long-horizon, multi-agent, and mediated by noisy...
arXiv AI
Sep 2
EDGE: Error Dependency Graph-Guided Multi-Error Attribution in Multi-Agent LLM Systems
EDGE is a framework that attributes multiple related errors in multi-agent large language model systems by constructing an error dependency graph from observed error events. It validates a reliable causal subset through counterfactual rollout and uses this inference graph to guide a two-stage LLM-as-judge detector for error attribution. Experiments on TRAIL and MAST demonstrate that EDGE improves category-level multi-error attribution across most models and settings, and that the graph aids explanation and repair analysis.
arXiv AI
Jul 14
Who&When Pro: Can LLMs Really Attribute Failures in AI Agents?
arXiv:2607. 09996v1 Announce Type: new Abstract: Automated failure attribution uses LLMs to identify where and why agentic systems fail.
arXiv AI
Aug 26
Adaptive Influence Graphs for Failure Attribution in Multi-Agent Systems
The paper introduces Adaptive Influence Graphs (AIGs), a two‑stage framework that first converts a failed trace into a structured graph and then navigates it to pinpoint the critical error in multi‑agent large language model systems. Experiments across multiple models demonstrate that richer trace representations and adaptive graph construction improve failure attribution, with AIGs achieving state‑of‑the‑art results on the Who&When benchmark. The study shows that both the diagnosing model and the way traces are represented and explored are crucial for accurate failure attribution.
arXiv Machine Learning
Jun 5
When Evidence is Sparse: Weakly Supervised Early Failure Alerting in Dialogs and LLM-Agent Trajectories
arXiv:2606. 05414v1 Announce Type: cross Abstract: Early failure alerting requires deciding, while a dialog or agent trajectory is still unfolding, whether to flag it as likely to fail.