arXiv AI
Jun 30
Diagnosing and Mitigating Context Rot in Long-horizon Search
arXiv:2606. 29718v1 Announce Type: cross Abstract: Extensive context has become the norm as Large Language Models (LLMs) are increasingly deployed in long-horizon tasks.
Diagnosing and Mitigating Context Rot in Long-horizon Search
Extensive context has become the norm as Large Language Models (LLMs) are increasingly deployed in long-horizon tasks. The concern that increasing context length degrades model capabilities, known as context rot, has become a central issue for these applications.
Related stories
Hugging Face Trending Papers
Jul 14
The Illusion of Robustness: Aggregate Accuracy Hides Prediction Flips under Task-Irrelevant Context
As large language models (LLMs) grow more capable, they are increasingly deployed in context-rich settings where task inputs are often accompanied by long, partially irrelevant context. In a controlled setting, we find that state-of-the-art models often appear robust to task-irrelevant context at the aggregate level: prepending it to benchmark questions causes little change in overall accuracy.
arXiv AI
Jun 30
MATCH: Modulating Attention via In-Context Retrieval for Long-Context Transformers
arXiv:2606. 29844v1 Announce Type: cross Abstract: The quadratic computational cost of traditional attention mechanisms poses a major bottleneck to the scalability and practical deployment of large language models (LLMs), particularly in long-context scenarios.
arXiv AI
Jun 4
100-LongBench: Are de facto Long-Context Benchmarks Literally Evaluating Long-Context Ability?
arXiv:2505. 19293v2 Announce Type: replace-cross Abstract: Long-context capability is considered one of the most important abilities of LLMs, as a truly long context-capable LLM enables users to effortlessly process many originally exhausting tasks -- e.
arXiv AI
6d ago
Information Abundance Paradox: Long-Context Training Undermines Parametric Knowledge
arXiv:2608. 12218v1 Announce Type: cross Abstract: Large language models are increasingly trained and deployed with long contexts that span documents, code repositories, and interaction histories.
arXiv AI
Jun 6
Dense Contexts Are Hard Contexts: Lexical Density Limits Effective Context in LLMs
arXiv:2606. 06203v1 Announce Type: cross Abstract: Input length and the position of relevant information are widely cited as the primary causes of degraded LLM long-context performance.
arXiv AI
Jun 26
Context Recycling for Long-Horizon LLM Inference
arXiv:2606. 26105v1 Announce Type: cross Abstract: Large language models (LLMs) exhibit strong capabilities in short-context reasoning but degrade in performance over long conversational horizons due to context window limitations and inefficient token usage.
arXiv AI
Jul 16
Not All Needles Are Found: How Fact Distribution and Don't Make It Up Prompts Shape Retrieval, Reasoning, and Hallucination in Long-Context LLMs
arXiv:2601. 02023v2 Announce Type: replace-cross Abstract: As Large Language Models (LLMs) increasingly utilize massive context windows as working memory for autonomous tasks, their reliability fluctuates significantly depending on how information is distributed in real-world corpora.
arXiv AI
Jun 2
ACON: Optimizing Context Compression for Long-horizon LLM Agents
arXiv:2510. 00615v3 Announce Type: replace Abstract: Large language models (LLMs) are increasingly deployed as agents in dynamic real-world environments, where success depends on maintaining precise records of actions and observations.
Hugging Face Trending Papers
Jul 9
Understanding Axes of Difficulty For Long Context Tasks Via PredicateLongBench
Large language models (LLMs) have demonstrated rapidly improving long-context capabilities, prompting a wave of benchmarks designed to evaluate them. However, existing long-context evaluations - from Needle-in-a-Haystack (NIAH) tests to more recent multi-hop reasoning and summarization tasks - predominantly measure average-case performance, and many are either saturated or lack robustness.
arXiv AI
Jul 10
Understanding Axes of Difficulty For Long Context Tasks Via PredicateLongBench
arXiv:2607. 08284v1 Announce Type: new Abstract: Large language models (LLMs) have demonstrated rapidly improving long-context capabilities, prompting a wave of benchmarks designed to evaluate them.
arXiv AI
Jul 14
Extending LLM Context via Associative Recurrent Memory
arXiv:2607. 11614v1 Announce Type: cross Abstract: Extending the context length of large language models (LLMs) is critical for many real-world applications, yet standard transformers remain constrained by quadratic compute and linear memory scaling.