← Back to all news
arXiv AI By Jeffrey Willette, Krishna C. Puvvada, Boris Ginsburg

Staying on Task: Testing the Foundations of Long-Horizon Agent Reliability

Read the original on arXiv AI →

The paper introduces Long-Transduction, a diagnostic framework designed to evaluate how well language models can maintain task fidelity during extended generation tasks that involve continuous reading, mutating, and outputting of context-dependent operations such as arithmetic, sorting, variable lookups, and table transformations. By independently varying local task complexity, input data formatting, and context length, the study isolates failure modes across these axes. Experiments on seven open-weight models reveal significant performance drops—62.8% when scaling context length from 4 to 128K, 36.5% with input format changes, and 39.9% with increased local task complexity—highlighting critical vulnerabilities in long-horizon agentic workflows.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Sep 3

How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making

The paper investigates why large language model (LLM) agents fail on long, multi‑step production workflows despite high benchmark success. By testing nine models (1.2 B–671 B parameters) across six task families and multiple horizons, the authors find that task success follows a geometric decay governed by a per‑step reliability that never reaches 1, leading to inevitable collapse for long horizons. The degradation is driven mainly by step count rather than context length, and the study quantifies a significant gap between benchmark and production performance, especially for agentic tool‑use tasks.

By Shubhra Mittal
llmsagentsbenchmarks
More like this →
arXiv AI
Sep 2

Parsing the Stream: A Live Trace Model for Long-Horizon Agents and Their Observers

The paper introduces a live trace model that incrementally folds an append‑only event ledger into typed run state, producing per‑consumer views for both human observers and the agent itself. Evaluations show that for observers, the compiled view reduces input tokens by 14–15× and cost by 5–7× while improving accuracy from 0.48 to 0.85–0.87. For agents, maintaining running statistics in per‑step state enables success on 120‑link sequential tasks where full‑context prompting fails, and a prompt‑level scratchpad matches the fold’s accuracy at lower cost.

By Egor Pakhomov, Erik Nijkamp
llmsagentsbenchmarks
More like this →