arXiv AI July 28, 2026 By Dhruvil S, Fenil Sojitra, Ravirajsinh Chauhan

Through the Bottleneck: How Multi-head Latent Attention Separates Content from Position in Language Models

arXiv:2607. 23054v1 Announce Type: cross Abstract: Multi-head Latent Attention (MLA), introduced in DeepSeek-V2, compresses key-value pairs through a shared low-rank bottleneck (cKV), achieving 81% KV-cache reduction during inference.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv Computation and Language

Sep 22

Garbage Attention in Large Language Models: BOS Sink Heads and Sink-aware Pruning

arXiv:2601.06787v2 Announce Type: replace Abstract: Large Language Models (LLMs) are known to contain significant redundancy, yet a systematic explanation for why certain components, particularly in...

By Jaewon Sok, Jewon Yeom, Seonghyeon Park, Jeongjae Park, Taesup Kim

llms efficiency

arXiv AI

Jul 24

Attention Degradation, Function Token Anchoring, and the Limits of Attention-Based Intervention in Large Language Models

arXiv:2607. 20524v1 Announce Type: new Abstract: Mean cross-positional attention degradation is widely reported in transformer interpretability, yet whether it causally limits contextual retrieval remains untested.

By Sagar Dangal, Manoj Shakya

llms safety

arXiv AI

Sep 3

Language Models Can Control Their Own Attention

The paper introduces Declarative Attention (DA), a protocol that lets language models explicitly declare which parts of their context to focus on during generation. By partitioning decoding into full-context, region-specific, and recent-output-only modes, the inference engine can skip large portions of the KV cache, dramatically reducing attended tokens. Experiments on 15 long-context tasks with off-the-shelf models show significant savings (52.0% and 31.1% reductions) with only modest accuracy drops that diminish as model size increases.

By Namgyu Ho, Huzama Ahmad, Woosung Koh, Se-Young Yun, Tal Schuster, Cicero Nogueira dos Santos

llms efficiency

arXiv AI

Jun 9

Contribution Weights: A Geometrical Analysis of Self-Attention Transformers

arXiv:2606. 07604v1 Announce Type: cross Abstract: Analyzing attention weights has become a standard approach for interpreting the information flow of Large Language Models (LLMs).

By Harry Jake Cunningham, Nicola Muca Cirone

llms safety

arXiv AI

Jul 22

A Controlled Study of Attention-Only Transformers

arXiv:2607. 18363v1 Announce Type: cross Abstract: Feed-forward networks hold two thirds of a transformer's non-embedding parameters, yet the architecture has not received a necessity test that controls parameters, compute, and depth at once.

By Henry Ndubuaku, Karen Mosoyan, Jakub Mroz, Noah Cylich, Satyajit Kumar, Parkirat Sandhu, Roman Shemet, Justin H Lee

llms rag

arXiv AI

Jun 11

The Structural Attention Tax: How Retrieval Format Hijacks In-Context Learning Independent of Content

arXiv:2606. 11198v1 Announce Type: cross Abstract: Retrieval-augmented generation (RAG) systems inject external knowledge to improve LLM outputs, yet the format of injected content -- distinct from its semantic relevance -- can independently distort the model's attention distribution.

By Yuqi Zhang, Di Zhang

llms rag benchmarks safety

One email a morning, machine-written

Related stories

Garbage Attention in Large Language Models: BOS Sink Heads and Sink-aware Pruning

Attention Degradation, Function Token Anchoring, and the Limits of Attention-Based Intervention in Large Language Models

Language Models Can Control Their Own Attention

Contribution Weights: A Geometrical Analysis of Self-Attention Transformers

A Controlled Study of Attention-Only Transformers

The Structural Attention Tax: How Retrieval Format Hijacks In-Context Learning Independent of Content