arXiv AI
Jun 30
Diagnosing and Mitigating Context Rot in Long-horizon Search
arXiv:2606. 29718v1 Announce Type: cross Abstract: Extensive context has become the norm as Large Language Models (LLMs) are increasingly deployed in long-horizon tasks.
Diagnosing and Mitigating Context Rot in Long-horizon Search
Extensive context has become the norm as Large Language Models (LLMs) are increasingly deployed in long-horizon tasks. The concern that increasing context length degrades model capabilities, known as context rot, has become a central issue for these applications.
Related stories
Hugging Face Trending Papers
Jul 14
The Illusion of Robustness: Aggregate Accuracy Hides Prediction Flips under Task-Irrelevant Context
As large language models (LLMs) grow more capable, they are increasingly deployed in context-rich settings where task inputs are often accompanied by long, partially irrelevant context. In a controlled setting, we find that state-of-the-art models often appear robust to task-irrelevant context at the aggregate level: prepending it to benchmark questions causes little change in overall accuracy.
arXiv Computation and Language
4d ago
LongHarness Bench: Stress-Testing Language Model Harnesses for Long-Context Reasoning
The paper introduces LongHarness Bench, a new benchmark designed to evaluate both the effectiveness and efficiency of language model harnesses for long-context reasoning. It features tasks that require diverse retrieval strategies—such as lexical search and semantic matching—and strategic, adaptive reasoning over global and local context, with only a small subset of the context being useful at each step. Evaluations across multiple state‑of‑the‑art models and harnesses show that even strong combinations achieve only 68% macro‑average accuracy, and that the same model can vary markedly in efficiency depending on the harness used.
arXiv AI
Jun 30
MATCH: Modulating Attention via In-Context Retrieval for Long-Context Transformers
arXiv:2606. 29844v1 Announce Type: cross Abstract: The quadratic computational cost of traditional attention mechanisms poses a major bottleneck to the scalability and practical deployment of large language models (LLMs), particularly in long-context scenarios.
arXiv AI
Jun 4
100-LongBench: Are de facto Long-Context Benchmarks Literally Evaluating Long-Context Ability?
arXiv:2505. 19293v2 Announce Type: replace-cross Abstract: Long-context capability is considered one of the most important abilities of LLMs, as a truly long context-capable LLM enables users to effortlessly process many originally exhausting tasks -- e.
arXiv AI
Aug 13
Information Abundance Paradox: Long-Context Training Undermines Parametric Knowledge
arXiv:2608. 12218v1 Announce Type: cross Abstract: Large language models are increasingly trained and deployed with long contexts that span documents, code repositories, and interaction histories.
arXiv Computation and Language
Sep 21
Boosting Deepresearch and LongContext Ability with Self-Generated Deepresearch Rollouts Traces
arXiv:2609.20844v1 Announce Type: new Abstract: Deepresearch (DR) agents interact with real-world web environments through multi-turn search and visit, causing their contexts to grow rapidly over tim...
arXiv Machine Learning
4d ago
Harnessing Large Language Models to Compile Task-Relevant Context into Bayesian Optimisation
arXiv:2609.36788v1 Announce Type: new Abstract: Incorporating rich task-relevant context, such as domain knowledge and external observations, is a key capability yet remains challenging for Bayesian...
arXiv AI
Jun 6
Dense Contexts Are Hard Contexts: Lexical Density Limits Effective Context in LLMs
arXiv:2606. 06203v1 Announce Type: cross Abstract: Input length and the position of relevant information are widely cited as the primary causes of degraded LLM long-context performance.
arXiv AI
Jun 26
Context Recycling for Long-Horizon LLM Inference
arXiv:2606. 26105v1 Announce Type: cross Abstract: Large language models (LLMs) exhibit strong capabilities in short-context reasoning but degrade in performance over long conversational horizons due to context window limitations and inefficient token usage.
arXiv AI
Jul 16
Not All Needles Are Found: How Fact Distribution and Don't Make It Up Prompts Shape Retrieval, Reasoning, and Hallucination in Long-Context LLMs
arXiv:2601. 02023v2 Announce Type: replace-cross Abstract: As Large Language Models (LLMs) increasingly utilize massive context windows as working memory for autonomous tasks, their reliability fluctuates significantly depending on how information is distributed in real-world corpora.
arXiv AI
Jun 2
ACON: Optimizing Context Compression for Long-horizon LLM Agents
arXiv:2510. 00615v3 Announce Type: replace Abstract: Large language models (LLMs) are increasingly deployed as agents in dynamic real-world environments, where success depends on maintaining precise records of actions and observations.