arXiv AI
Jul 15
Tracing Agentic Failure from the Flow of Success
arXiv:2607. 12747v1 Announce Type: new Abstract: Failure attribution for LLM-based agentic systems, i.
Do Agents Know When They Succeed? Calibrating Agent Confidence from Internal Representations
The paper examines whether internal representations of agentic systems can better indicate task success than traditional confidence measures. It introduces two methods—Latent Trajectory Dynamics (LTD) and Action Representation Probe (ARP)—that analyze changes in residual-stream representations and action-level representations, respectively. Experiments on Bash, SQL, and Python benchmarks with Qwen and DeepSeek models show these methods outperform conventional surface-level and sequence-based calibration baselines, offering a zero‑overhead reliability monitor without prompt changes or multiple rollouts.
Related stories
arXiv Machine Learning
Sep 1
Last Step Matters: Early Uncertainty Cannot Predict Failure in Long-Horizon Agents
arXiv:2608.29685v1 Announce Type: new Abstract: Early failure prediction is important for long-horizon agents, as it enables timely intervention and can reduce inference and tool-use costs. Uncertain...
arXiv AI
Jun 2
AgentProcessBench: Diagnosing Step-Level Process Quality in Tool-Using Agents
arXiv:2603. 14465v2 Announce Type: replace Abstract: While Large Language Models (LLMs) have evolved into tool-using agents, they remain brittle in long-horizon interactions.
arXiv AI
Sep 1
AgentRx: Diagnosing AI Agent Failures from Execution Trajectories
arXiv:2602.02475v2 Announce Type: replace Abstract: AI agents often fail in ways that are difficult to localize because executions are probabilistic, long-horizon, multi-agent, and mediated by noisy...
arXiv AI
Aug 18
From Sequence to Structure: Relational Uncertainty Propagation for LLM Agents
The paper introduces RUPA, a trajectory‑level uncertainty quantification framework for large language model agents. RUPA models an agent’s execution as a directed graph of reasoning states, tool interactions, and environment feedback, then propagates uncertainty across this graph to capture long‑range dependencies. Experiments on benchmarks such as τ‑2, Terminal‑Bench‑2, and GAIA show that RUPA outperforms existing methods, enabling earlier failure detection and more reliable agent execution.
arXiv AI
Jul 8
Beyond Reactivity: Measuring Proactive Problem Solving in LLM Agents
arXiv:2510. 19771v4 Announce Type: replace Abstract: LLM-based agents are increasingly moving towards proactivity: rather than awaiting instruction, they exercise agency to anticipate user needs and solve them autonomously.
arXiv AI
3d ago
Verifiable Hidden Dynamics Play: Generating Agentic RL Environments from Solved Mechanisms
The paper introduces VHD-Play, a pipeline that first samples and solves a mathematical model before generating agentic reinforcement learning environments, ensuring that dynamics and evaluation are aligned from the outset. This approach yields 3,300 diverse environments at a low cost and significantly improves the performance of a large language‑model agent (Qwen3.6‑35B‑A3B) across multiple diagnostic families and external benchmarks. The study demonstrates that stateful interaction is a key factor in learning gains and that scaling the training substrate can further enhance performance.
arXiv Machine Learning
5d ago
TRACES: Proactive Safety Auditing for Multi-Turn LLM Agents via Trajectory-State Modeling
arXiv:2605.27690v2 Announce Type: replace-cross Abstract: LLM agents increasingly operate through multi-turn tool use and environment interaction, where safety risks often emerge from intermediate st...
arXiv AI
Jun 2
From Features to Actions: Explainability in Traditional and Agentic AI Systems
arXiv:2602. 06841v4 Announce Type: replace Abstract: Over the last decade, Explainable AI has primarily focused on interpreting individual model predictions, producing post-hoc explanations that relate inputs to outputs under a fixed decision structure.
arXiv AI
Jun 9
ATM: Action-Consistency Transfer Matrix for Diagnosing and Improving Latent World Models
arXiv:2606. 09028v1 Announce Type: cross Abstract: Latent world models are increasingly used for control and goal-conditioned planning, yet assessing whether their learned representations are useful for planning usually requires slow, planner-coupled simulator evaluation with CEM or similar planners.
arXiv Machine Learning
Sep 7
How to Speculate about Uncertainty in Agentic Coding? A Draft-Model Gate Method
The paper introduces Speculative Uncertainty (SU), a technique that infers a failure likelihood for black‑box LLM agents by evaluating their generated token sequences with a lightweight draft model, without needing internal model details. SU extracts phase‑aware features from reasoning and action spans, calibrates them against verifiable outcomes, and produces a failure‑likelihood score usable by downstream policies. Applying a pre‑execution veto gate based on SU to software‑engineering agents such as Qwen3‑Coder‑480B and Claude 3.5 Sonnet reduced execution error rates by 6‑8 percentage points and token costs by 14‑19 %, while maintaining performance on out‑of‑distribution benchmarks and across different agent models.
arXiv AI
Aug 7
TRAJDEBUG: Tracing Error Lifecycle to Identify Critical Failures in Long-Horizon Agent Trajectories
arXiv:2608. 06346v1 Announce Type: new Abstract: LLM-based agentic systems have shown remarkable capabilities in complex domains, while suffering from cascading errors and difficulty in debugging.