The paper introduces a method for monitoring web agents without relying on internal signals such as token logits. It proposes two observable trajectory representations—Macro features that capture cross‑step agent–environment interactions, and Micro features that assess consistency of intention, action, and expected state change via repeated black‑box queries. By labeling the first uncorrected critical error that leads to final failure as a key‑step boundary, the approach preserves valid early prefixes of failed trajectories and achieves risk prediction performance competitive with internal‑signal baselines across WebArena‑Lite and Online Mind2Web benchmarks.
arXiv:2605.27690v2 Announce Type: replace-cross Abstract: LLM agents increasingly operate through multi-turn tool use and environment interaction, where safety risks often emerge from intermediate st...
arXiv:2610.09115v1 Announce Type: cross Abstract: Steering an LLM agent means deciding whether to correct it, at which step, and with which mechanism. Uncertainty is often used to decide when to corr...
arXiv:2602.02475v2 Announce Type: replace Abstract: AI agents often fail in ways that are difficult to localize because executions are probabilistic, long-horizon, multi-agent, and mediated by noisy...
The paper introduces SOAP, a method for attributing failures in large language model agents without requiring step-level labels or a separate set of successful trajectories. SOAP builds a spectral reference space from LLM representations to assign base error scores to each step, then uses attention-based soft dependencies to propagate downstream error signals back to their likely upstream causes. Experiments demonstrate that SOAP outperforms existing baselines in accurately identifying decisive steps responsible for task failures.
arXiv:2608. 06346v1 Announce Type: new Abstract: LLM-based agentic systems have shown remarkable capabilities in complex domains, while suffering from cascading errors and difficulty in debugging.
The paper introduces ontological trust, a task‑conditioned property of trajectory prefixes, and presents RGE, an online monitor that decomposes trust into Role, Goal, and Evidence. RGE uses LLMs only for structured task and step representations, while trust updates and interventions are deterministic, producing a replayable and auditable trust trajectory. Evaluated on a cross‑domain corpus, RGE outperforms rule‑, judge‑, and shield‑style baselines, achieving over 93% Drift F1 and maintaining high benign coverage.
arXiv:2606. 05414v1 Announce Type: cross Abstract: Early failure alerting requires deciding, while a dialog or agent trajectory is still unfolding, whether to flag it as likely to fail.
The paper examines whether internal representations of agentic systems can better indicate task success than traditional confidence measures. It introduces two methods—Latent Trajectory Dynamics (LTD) and Action Representation Probe (ARP)—that analyze changes in residual-stream representations and action-level representations, respectively. Experiments on Bash, SQL, and Python benchmarks with Qwen and DeepSeek models show these methods outperform conventional surface-level and sequence-based calibration baselines, offering a zero‑overhead reliability monitor without prompt changes or multiple rollouts.
arXiv:2608.29685v1 Announce Type: new Abstract: Early failure prediction is important for long-horizon agents, as it enables timely intervention and can reduce inference and tool-use costs. Uncertain...
arXiv:2606. 03467v1 Announce Type: new Abstract: LLM-based multi-agent systems exhibit remarkable collaborative capabilities in complex multi-step tasks.
The paper introduces Interaction-Effect Propagation (IE-Prop), a model that estimates error risk in large language model (LLM) based multi‑agent systems by conditioning on whether upstream agents are correct or incorrect. IE‑Prop learns separate effects for each sender‑receiver interaction, combines them using the sender’s estimated error risk, and recursively propagates these risks through the execution graph. Experiments on three architectures and benchmarks show that IE‑Prop improves final‑node AUROC by about 0.05 points over baselines and that multi‑step risk tracking adds an additional 0.03 AUROC in sequential systems.
arXiv:2607. 26820v1 Announce Type: new Abstract: As large language models (LLMs) evolve from standalone assistants into autonomous agents, ensuring their safety requires shifting beyond pointwise risk assessment to understand how risks emerge and unfold over long-horizon trajectories.