arXiv Machine Learning By Konstantin Grotov, Valentin Malykh

How to Speculate about Uncertainty in Agentic Coding? A Draft-Model Gate Method

Read the original on arXiv Machine Learning →

The paper introduces Speculative Uncertainty (SU), a technique that infers a failure likelihood for black‑box LLM agents by evaluating their generated token sequences with a lightweight draft model, without needing internal model details. SU extracts phase‑aware features from reasoning and action spans, calibrates them against verifiable outcomes, and produces a failure‑likelihood score usable by downstream policies. Applying a pre‑execution veto gate based on SU to software‑engineering agents such as Qwen3‑Coder‑480B and Claude 3.5 Sonnet reduced execution error rates by 6‑8 percentage points and token costs by 14‑19 %, while maintaining performance on out‑of‑distribution benchmarks and across different agent models.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv AI
Oct 16

AgentPerturb: Evaluating LLM Agent Robustness under Controlled Interaction Perturbations

AgentPerturb introduces a systematic framework to evaluate large language model agents under controlled interaction perturbations, addressing the lack of unified evaluation methods for noisy real‑world tasks. The study quantifies robustness across frontier models, identifies task‑specific vulnerabilities, and shows that agents often fail to detect altered inputs, propagating errors and incurring higher computational costs. The authors provide a benchmark‑construction pipeline and plan to release code and data to support future robustness‑oriented training.

By Yuanjian Xu, Jinyi Han, Jianing Hao, Yuxuan Sun, Fanyang Lu, Richeng Xuan, Guang Zhang, Zhichao Hu
arXiv Computation and Language
Oct 12

ARTIS: Agentic Risk-Aware Test-Time Scaling via Iterative Simulation

The paper introduces ARTIS, a framework that separates exploration from commitment by using iterative simulation to test large language model actions before real-world execution. This approach extends inference-time computation to boost action-level reliability and robustness while avoiding environmental risk. The authors also highlight that standard LLM simulators miss rare, high-impact failures, and propose a risk‑aware simulator that focuses on fidelity for failure‑inducing actions, showing significant reliability gains across agentic benchmarks.

By Xingshan Zeng, Lingzhi Wang, Weiwen Liu, Liangyou Li, Yasheng Wang, Lifeng Shang, Xin Jiang, Qun Liu
arXiv AI
Sep 11

Do Agents Know When They Succeed? Calibrating Agent Confidence from Internal Representations

The paper examines whether internal representations of agentic systems can better indicate task success than traditional confidence measures. It introduces two methods—Latent Trajectory Dynamics (LTD) and Action Representation Probe (ARP)—that analyze changes in residual-stream representations and action-level representations, respectively. Experiments on Bash, SQL, and Python benchmarks with Qwen and DeepSeek models show these methods outperform conventional surface-level and sequence-based calibration baselines, offering a zero‑overhead reliability monitor without prompt changes or multiple rollouts.

By Priyanka Mary Mammen, Emil Joswin, Srujananjali Medicherla
arXiv AI
Oct 5

LEAP: Learning Efficient Action Proposals For LLM Agents

LEAP: Learning Efficient Action Proposals For LLM Agents proposes a method to speed up large language model agents by training a small 0.6B drafter to predict target actions accurately. The approach uses a latency framework that balances drafting, verification, and execution costs, achieving up to 60% faster end‑to‑end wall clock time without reducing task success. LEAP can be online trained, eliminating the need for prior trace collection and making it practical for real‑world deployment.

By Zhen Xu, Qizheng Zhang, Gerry Wan, Shang Zhu, Ce Zhang
arXiv AI
Aug 7

TRAJDEBUG: Tracing Error Lifecycle to Identify Critical Failures in Long-Horizon Agent Trajectories

arXiv:2608. 06346v1 Announce Type: new Abstract: LLM-based agentic systems have shown remarkable capabilities in complex domains, while suffering from cascading errors and difficulty in debugging.

By Yunjia Qi, Zehua Yin, Xintong Shi, Hao Peng, Songyuanyi Lu, Yixian Liu, Richeng Xuan, Yuhong Liu, Zhichao Hu, Xiaozhi Wang, Lei Hou, Bin Xu, Juanzi Li