arXiv AI September 3, 2026 By Namgyu Ho, Huzama Ahmad, Woosung Koh, Se-Young Yun, Tal Schuster, Cicero Nogueira dos Santos

Language Models Can Control Their Own Attention

Read the original on arXiv AI →

The paper introduces Declarative Attention (DA), a protocol that lets language models explicitly declare which parts of their context to focus on during generation. By partitioning decoding into full-context, region-specific, and recent-output-only modes, the inference engine can skip large portions of the KV cache, dramatically reducing attended tokens. Experiments on 15 long-context tasks with off-the-shelf models show significant savings (52.0% and 31.1% reductions) with only modest accuracy drops that diminish as model size increases.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI

Jun 18

Dual Dimensionality for Local and Global Attention

arXiv:2606. 18587v1 Announce Type: cross Abstract: Decoder-only Transformers compute attention over the KV cache of preceding tokens.

By Zhiyuan Wang, Xuan Luo, Sirui Zeng, Xifeng Yan

llms fine-tuning efficiency

arXiv AI

Jul 7

Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling

arXiv:2607. 02980v1 Announce Type: cross Abstract: Scaling modern large language models (LLMs) to long contexts is limited by the quadratic computation cost, and poor length extrapolation of dense attention.

By Xiang Hu, Xinyu Wei, Hao Gu, Minshen Zhang, Tian Liang, Huayang Li, Lei Zhu, Yan Wang, Sirui Han, Yushi Bai, Kewei Tu, Haitao Mi, Leo Liang

llms

arXiv Machine Learning

Jun 29

Simplified Sparse Attention via Gist Tokens

arXiv:2604. 20920v2 Announce Type: replace Abstract: Sparse attention can reduce the cost of long-context inference, but most variants introduce new architectural components.

By Yuzhen Mao, Michael Y. Li, Emily B. Fox

llms rag efficiency

arXiv AI

Jun 9

Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps

arXiv:2605. 16928v2 Announce Type: replace-cross Abstract: Long-context inference in large language models is bottlenecked by the quadratic cost of full attention.

By Yanke Zhou, Yiduo Li, Hanlin Tang, Maohua Li, Kan Liu, Tao Lan, Lin Qu, Yuan Yao, Xiaoxing Ma

llms efficiency benchmarks

arXiv Machine Learning

Jul 27

RIS-Kernel: A Model-Agnostic Architecture for Long-Context LLM Inference via Sparse Attention

arXiv:2607. 21927v1 Announce Type: new Abstract: Full self-attention in large language models scales as O(N^2), which limits long-context document analysis to 65,536 tokens and requires costly GPU clusters.

By Anderson R. Santos

llms

arXiv AI

Jul 23

LISA: Linear-Indexed Sparse Attention for Efficient Long-Context Reasoning

arXiv:2607. 19358v1 Announce Type: new Abstract: Recent advances in long chain-of-thought reasoning models such as DeepSeek-R1 have led to increasingly longer inference context lengths under the test-time scaling paradigm.

By Yu Zhao, Zekun Zhang, Fan Jiang, Bo Zeng, Linlong Xu, Shimin Shan, Yu Liu, Longyue Wang, Weihua Luo

efficiency benchmarks

One email a morning, machine-written

Related stories

Dual Dimensionality for Local and Global Attention

Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling

Simplified Sparse Attention via Gist Tokens

Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps

RIS-Kernel: A Model-Agnostic Architecture for Long-Context LLM Inference via Sparse Attention

LISA: Linear-Indexed Sparse Attention for Efficient Long-Context Reasoning