arXiv AI July 24, 2026 By Sagar Dangal, Manoj Shakya

Attention Degradation, Function Token Anchoring, and the Limits of Attention-Based Intervention in Large Language Models

arXiv:2607. 20524v1 Announce Type: new Abstract: Mean cross-positional attention degradation is widely reported in transformer interpretability, yet whether it causally limits contextual retrieval remains untested.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI

Jun 11

The Structural Attention Tax: How Retrieval Format Hijacks In-Context Learning Independent of Content

arXiv:2606. 11198v1 Announce Type: cross Abstract: Retrieval-augmented generation (RAG) systems inject external knowledge to improve LLM outputs, yet the format of injected content -- distinct from its semantic relevance -- can independently distort the model's attention distribution.

By Yuqi Zhang, Di Zhang

llms rag benchmarks safety

arXiv Machine Learning

Aug 7

QEvict: Recoverable Quantized KV Eviction for Attention-Drift-Robust Long-Context Decoding

arXiv:2608. 05326v1 Announce Type: new Abstract: Autoregressive large language model inference is increasingly constrained by the memory footprint of the Key-Value (KV) cache.

By Ayushman Garg, Akshita Gupta, Shaswata Bhattacharya, Abhishek Gupta, Sandeep Kumar, Manoj Kumar

llms efficiency benchmarks

arXiv Computation and Language

3d ago

RoPE at the End of Its Rope? Theory, Diagnosis, and Mitigation of Long-Context Failures

arXiv:2609.39929v1 Announce Type: cross Abstract: Long-context failures of RoPE-based language models can arise from RoPE's intrinsic tradeoff between maintaining stable token preferences and disting...

By Yuyang Wu, Yufeng Du, Hao Peng

llms benchmarks

arXiv AI

Sep 3

Language Models Can Control Their Own Attention

The paper introduces Declarative Attention (DA), a protocol that lets language models explicitly declare which parts of their context to focus on during generation. By partitioning decoding into full-context, region-specific, and recent-output-only modes, the inference engine can skip large portions of the KV cache, dramatically reducing attended tokens. Experiments on 15 long-context tasks with off-the-shelf models show significant savings (52.0% and 31.1% reductions) with only modest accuracy drops that diminish as model size increases.

By Namgyu Ho, Huzama Ahmad, Woosung Koh, Se-Young Yun, Tal Schuster, Cicero Nogueira dos Santos

llms efficiency

arXiv AI

Jul 28

Through the Bottleneck: How Multi-head Latent Attention Separates Content from Position in Language Models

arXiv:2607. 23054v1 Announce Type: cross Abstract: Multi-head Latent Attention (MLA), introduced in DeepSeek-V2, compresses key-value pairs through a shared low-rank bottleneck (cKV), achieving 81% KV-cache reduction during inference.

By Dhruvil S, Fenil Sojitra, Ravirajsinh Chauhan

llms fine-tuning safety

Hugging Face Trending Papers

3d ago

RoPE at the End of Its Rope? Theory, Diagnosis, and Mitigation of Long-Context Failures

Long-context failures of RoPE-based language models can arise from RoPE's intrinsic tradeoff between maintaining stable token preferences and distinguishing nearby positions. Determining which weaknes...

llms benchmarks

One email a morning, machine-written

Related stories

The Structural Attention Tax: How Retrieval Format Hijacks In-Context Learning Independent of Content

QEvict: Recoverable Quantized KV Eviction for Attention-Drift-Robust Long-Context Decoding

RoPE at the End of Its Rope? Theory, Diagnosis, and Mitigation of Long-Context Failures

Language Models Can Control Their Own Attention

Through the Bottleneck: How Multi-head Latent Attention Separates Content from Position in Language Models

RoPE at the End of Its Rope? Theory, Diagnosis, and Mitigation of Long-Context Failures