arXiv AI October 19, 2026 By Zhenxu Tian, Kebin Liu, Zhengwu Yang, Yi Su, Qingqing Dang, Yanlin Sha, Kaipeng Deng, Dianhai Yu, Yanjun Ma, Juntao Li, Min Zhang

Prism Attention: Proposal-Refined Index Sharing Mechanism for Efficient LLMs Inference

Read the original on arXiv AI →

Prism Attention introduces a proposal‑refined sparse attention framework that replaces static cross‑layer index reuse with a coarse‑to‑fine strategy, allowing layer‑wise adaptability while maintaining efficiency. The method leverages the observation that high‑attention tokens remain largely stable across layers, with changes concentrated near the boundary of previously selected regions. Experiments demonstrate state‑of‑the‑art performance on reasoning and long‑context benchmarks, achieving a 3.4× decoding speedup over FlashAttention at 128K context length.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI

Sep 1

Kascade: A Practical Sparse Attention Method for Long-Context LLM Inference

arXiv:2512.16391v2 Announce Type: replace-cross Abstract: Attention is the dominant source of latency during long-context LLM inference, an increasingly popular workload with reasoning models and RAG...

By Dhruv Deshmukh, Saurabh Goyal, Nipun Kwatra, Ramachandran Ramjee

llms rag benchmarks

arXiv AI

Jun 9

Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps

arXiv:2605. 16928v2 Announce Type: replace-cross Abstract: Long-context inference in large language models is bottlenecked by the quadratic cost of full attention.

By Yanke Zhou, Yiduo Li, Hanlin Tang, Maohua Li, Kan Liu, Tao Lan, Lin Qu, Yuan Yao, Xiaoxing Ma

llms efficiency benchmarks

arXiv Machine Learning

Jun 5

You Only Index Once: Cross-Layer Sparse Attention with Shared Routing

arXiv:2606. 06467v1 Announce Type: cross Abstract: Long-context inference in modern LLMs is increasingly constrained by decoding efficiency, especially in reasoning-heavy settings where models generate long intermediate chains of thought.

By Yutao Sun, Yanqi Zhang, Li Dong, Jianyong Wang, Furu Wei

llms efficiency benchmarks

arXiv Computation and Language

5d ago

Titan-Llama: Retrofitting Pre-trained LLMs with External Neural Memory Modules

arXiv:2610.19210v1 Announce Type: new Abstract: Transformers are efficient to pre-train at scale because attention permits parallel computation across sequence positions, but long-context inference r...

By Sarah Pan, Ali Behrouz

llms benchmarks

arXiv Machine Learning

Sep 23

CompKV: Compensation-Aware KV Selection for Long-Context LLM Inference

CompKV introduces a compensation‑aware sparse attention framework for long‑context LLM inference. It partitions tokens into blocks and optimizes token selection to minimize the error introduced by block‑level mean compensation, using compact block‑level statistics. Experiments on RULER and LongBench‑Pro show CompKV outperforms other sparse baselines and achieves up to a 6.85× speedup over full attention.

By Zhen Huang, Ruizhe Yao, Danyi Liu, Xinrui Chen, Shuwei Li, Siru Zhong, Zijian Cao, Yushan Lai, Mingming Guo, Weijie Zheng, Haohuan Fu

llms efficiency

arXiv Machine Learning

Jun 29

Simplified Sparse Attention via Gist Tokens

arXiv:2604. 20920v2 Announce Type: replace Abstract: Sparse attention can reduce the cost of long-context inference, but most variants introduce new architectural components.

By Yuzhen Mao, Michael Y. Li, Emily B. Fox

llms rag efficiency

One email a morning, machine-written

Related stories

Kascade: A Practical Sparse Attention Method for Long-Context LLM Inference

Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps

You Only Index Once: Cross-Layer Sparse Attention with Shared Routing

Titan-Llama: Retrofitting Pre-trained LLMs with External Neural Memory Modules

CompKV: Compensation-Aware KV Selection for Long-Context LLM Inference

Simplified Sparse Attention via Gist Tokens