arXiv Machine Learning October 8, 2026 By Yao Fu, Cyrus Chang, Ritchie Zhao, Bryce Long, Yueying Li, Mahdi Kamani, Samkit Jain, Rahul Raman, Tara Safavi, Shreya Gupta, Parsa Ashrafi Fashi, Minseok Lee, Julien Demouth, Bita Darvish Rouhani

SPIN: Shadow Predictive Indexer for Sparse Attention

Read the original on arXiv Machine Learning →

SPIN (Shadow Predictive Indexer) is a new method for sparse attention that reduces the overhead of scoring the entire key-value (KV) cache during decoding. By using lightweight, history-based predictions to identify important KV blocks, SPIN avoids full cache scoring at each step and treats KV blocks and speculative decoding as core design elements. Experiments on long-context and agentic benchmarks show 30‑40% sparsity with maintained task quality, and in vLLM serving it boosts throughput by up to 14.9% while cutting median inter‑token latency by up to 13.2%.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Machine Learning

Jun 2

Vegas: Self-Speculative Decoding with Verification-Guided Sparse Attention

arXiv:2602. 07223v2 Announce Type: replace Abstract: Long-context large language model (LLM) inference has become the norm for today's AI applications.

By Yikang Yue, Yuqi Xue, Jian Huang

llms efficiency benchmarks

arXiv Machine Learning

Jul 1

RaBitQCache: Rotated Binary Quantization for KVCache in Long Context LLM Inference

arXiv:2606. 31519v1 Announce Type: new Abstract: Long-context Large Language Model inference is severely bottlenecked by the massive Key-Value (KV) cache, yet existing sparse attention methods often suffer from static fixed-budget (Top-k) retrieval or rely on proxy scores that are computationally expensive and biased.

By Wenhao Li, Jinhao Dong, Hailin Zhang, Wenhang Shi, Wei Lu, Xiaoyong Du

llms efficiency benchmarks

arXiv Machine Learning

Jun 5

You Only Index Once: Cross-Layer Sparse Attention with Shared Routing

arXiv:2606. 06467v1 Announce Type: cross Abstract: Long-context inference in modern LLMs is increasingly constrained by decoding efficiency, especially in reasoning-heavy settings where models generate long intermediate chains of thought.

By Yutao Sun, Yanqi Zhang, Li Dong, Jianyong Wang, Furu Wei

llms efficiency benchmarks

arXiv Computation and Language

Sep 23

HySparse2: Hybrid Sparse Attention with Two-Level KV Sharing

arXiv:2609.26368v1 Announce Type: new Abstract: Long-horizon and multi-turn agents typically generate short actions and process long observations from tools and environments. This growing context dem...

By Jianyu Wei, Yizhao Gao, Qihao Zhang, Shimao Chen, Zhengju Tang, Yu Cheng, Shengjie Zhou, Zihan Jiang, Yifan Song, Hailin Zhang, Liang Zhao, Bo Yang, Gang Wang, Shijie Cao, Fuli Luo

agents efficiency

arXiv AI

Oct 7

AttSVD:Prompt-Adaptive Low-Rank KV Cache Compression via Attention-Guided SVD

arXiv:2610.06927v1 Announce Type: cross Abstract: The key-value (KV) cache of autoregressive transformers grows linearly with context length and dominates memory at long context. Most training-free r...

By Sara Abdali, Jongwoo Ko, Pashmina Cameron

llms agents efficiency benchmarks safety

arXiv Machine Learning

Oct 15

FOLD: Position-Free Far Memory with Local Order for Long-Context Speculative Decoding

arXiv:2610.20392v1 Announce Type: cross Abstract: Speculative decoding accelerates LLM generation through lightweight drafting and parallel verification. It is particularly promising for long inputs,...

By Shuxuan Li, Jialiang Kang, Yanghao Zhou, Ziye Zhu, Xiao Yang

llms benchmarks

One email a morning, machine-written

Related stories

Vegas: Self-Speculative Decoding with Verification-Guided Sparse Attention

RaBitQCache: Rotated Binary Quantization for KVCache in Long Context LLM Inference

You Only Index Once: Cross-Layer Sparse Attention with Shared Routing

HySparse2: Hybrid Sparse Attention with Two-Level KV Sharing

AttSVD:Prompt-Adaptive Low-Rank KV Cache Compression via Attention-Guided SVD

FOLD: Position-Free Far Memory with Local Order for Long-Context Speculative Decoding