The paper introduces Long-Transduction, a diagnostic framework designed to evaluate how well language models can maintain task fidelity during extended generation tasks that involve continuous reading, mutating, and outputting of context-dependent operations such as arithmetic, sorting, variable lookups, and table transformations. By independently varying local task complexity, input data formatting, and context length, the study isolates failure modes across these axes. Experiments on seven open-weight models reveal significant performance drops—62.8% when scaling context length from 4 to 128K, 36.5% with input format changes, and 39.9% with increased local task complexity—highlighting critical vulnerabilities in long-horizon agentic workflows.
arXiv:2607. 27283v1 Announce Type: new Abstract: Long-horizon benchmarks often show that agents fail more as tasks become longer.
The paper investigates why large language model (LLM) agents fail on long, multi‑step production workflows despite high benchmark success. By testing nine models (1.2 B–671 B parameters) across six task families and multiple horizons, the authors find that task success follows a geometric decay governed by a per‑step reliability that never reaches 1, leading to inevitable collapse for long horizons. The degradation is driven mainly by step count rather than context length, and the study quantifies a significant gap between benchmark and production performance, especially for agentic tool‑use tasks.
The paper introduces a live trace model that incrementally folds an append‑only event ledger into typed run state, producing per‑consumer views for both human observers and the agent itself. Evaluations show that for observers, the compiled view reduces input tokens by 14–15× and cost by 5–7× while improving accuracy from 0.48 to 0.85–0.87. For agents, maintaining running statistics in per‑step state enables success on 120‑link sequential tasks where full‑context prompting fails, and a prompt‑level scratchpad matches the fold’s accuracy at lower cost.
arXiv:2607. 11388v1 Announce Type: new Abstract: Recent advances in large language models (LLMs) and vision-language models (VLMs) have enabled increasingly capable digital agents for computer use.
arXiv:2606. 07808v1 Announce Type: new Abstract: Reasoning language models deployed in agentic workflows must follow an instruction hierarchy: when instructions from different sources conflict, the model should obey the highest-privilege applicable instruction.
arXiv:2606. 01365v1 Announce Type: new Abstract: Tool-using multi-agent large language model (LLM) systems spend computation through model tokens, tool calls, retries, and code execution before producing an answer.
LongDS-Bench is a new benchmark for evaluating long-horizon, multi-turn data analysis by agents, featuring 68 tasks derived from real-world Kaggle notebooks that span 2,225 turns across six domains such as Geoscience, Business, and Education. The benchmark focuses on agents’ ability to maintain, update, restore, and compose evolving analytical states, with tasks designed around state-evolution patterns like counterfactual perturbation, rollback, and multi-state composition, and an average dependency span of 11.3 turns. Evaluation of five state-of-the-art models shows that the best model achieves only 48.45% average accuracy, with performance dropping nearly 47 points from early to late turns and long-horizon errors accounting for 52%–69% of failures, indicating that maintaining a correct analytical state is the key bottleneck.
The paper proposes a hierarchical architecture for long-horizon language‑model agents that must operate over days or weeks without forgetting. It introduces three key components: time‑scale levels that store bounded summaries, a clocked tick as the basic action unit, and cascaded intelligence that escalates tasks to more capable models only after review failures. A ten‑day experiment demonstrated that the agent maintained continuity across context resets, adapted its behavior based on early knowledge, and identified where learned components could be integrated.
arXiv:2608.29128v1 Announce Type: new Abstract: Tool-using agents are commonly evaluated by a single bit: whether an end-to-end workflow completed. This metric fails to distinguish failures that matt...
The paper introduces Traverse, a benchmark of 2,518 agent trajectories and 6,967 annotated mistakes across software engineering, computer use, and science tasks, revealing that failures often go unrecovered and can cause irreversible harm before a run is deemed successful. It shows that human judges struggle to detect the first mistake in most runs, while a 4‑billion‑parameter verifier called Scout can locate failures more effectively and improve task success when used to select among candidate runs. The study demonstrates that making failure detection inexpensive and reliable can enable long‑horizon agents to learn from their own mistakes and increase trustworthiness in autonomous AI.
arXiv:2609.08589v1 Announce Type: cross Abstract: Recent large language models can emit task-progress signals that agent frameworks use to decide whether a task should continue or stop, yet whether a...
The paper introduces a benchmark for evaluating large language models (LLMs) on long‑horizon state tracking by having them compute the MD5 hash through 196 dependent tool calls across 64 rounds, carrying four 32‑bit words in context. It shows that a mixture‑of‑experts LLM can maintain the full state and produce correct digests in most runs, even when all primitive tools are replaced by another LLM. The study isolates state‑tracking difficulty from instruction interpretation and identifies key factors—contextual reasoning and worker voting—that enable success.