arXiv AI
The Unreliable Progress Bar: Can LLM Agents Reliably Report Task Progress Throughout Execution?
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Oct 15
Long-Horizon-Terminal-Bench 2.0: Automating Diverse Task Construction and Measuring Sustained Agent Progress
arXiv:2610.14146v1 Announce Type: new Abstract: Long-horizon terminal benchmarks measure whether an agent can sustain progress through extended workflows with process rewards, not only whether its fi...
arXiv AI
Oct 26
Async-RBench: Benchmarking Asynchronous Task Execution in LLM Agents
Async-RBench is a new benchmark that tests how large language model agents handle asynchronous task execution. It includes 200 executable tasks across eight scenarios, with a scenario controller that delivers results based on task-state conditions and an Event Response Contract that defines required changes, preservation, avoidance, and re-verification. The benchmark introduces the Dynamic Replanning Score (DRS) to evaluate both the response process and outcome, and finds that agents generally perform worse with asynchronous delivery compared to batched delivery, highlighting challenges in reliable asynchronous task handling.
arXiv AI
Oct 15
AM-Bench: Understanding Agents under Task Underspecification across Models and Harnesses
arXiv:2610.18090v1 Announce Type: new Abstract: Ambiguous or incomplete task specifications complicate both execution and clarification for language-model agents. Yet agents' responses to these infor...
arXiv AI
Oct 15
SkillHorizon: Constructing Long-Horizon Agent Tasks through Skill Composition
arXiv:2610.14966v1 Announce Type: new Abstract: Long-horizon agents must complete workflows in which later activities depend on earlier outputs and changes to shared state. A key challenge for evalua...
arXiv Computation and Language
Oct 22
Bridging Foundational and Agentic Capabilities of LLMs
The paper introduces a framework that links the foundational capabilities of large language models (LLMs) to their agentic performance by measuring operation-level gaps. It constructs foundational profiles from benchmark data and operation profiles from real interaction trajectories, then quantifies how much observed operation performance deviates from predictions based on foundational capabilities. Experiments on fifteen model services in the τ²-Bench demonstrate that foundational profiles alone only partially predict agentic outcomes, while incorporating operation profiles significantly improves task-score predictions and reveals consistent execution gaps across task subsets.
arXiv AI
Sep 3
How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making
The paper investigates why large language model (LLM) agents fail on long, multi‑step production workflows despite high benchmark success. By testing nine models (1.2 B–671 B parameters) across six task families and multiple horizons, the authors find that task success follows a geometric decay governed by a per‑step reliability that never reaches 1, leading to inevitable collapse for long horizons. The degradation is driven mainly by step count rather than context length, and the study quantifies a significant gap between benchmark and production performance, especially for agentic tool‑use tasks.