arXiv AI November 6, 2026 By Longwei Zou, Lin Zhong

KVBuffer: IO-aware Serving for Linear Attention

KVBuffer is an IO‑aware serving mechanism designed to improve the efficiency of linear attention models. By buffering recent keys and values, it enables chunkwise decoding, parallel speculative decoding, and direct attention computation for short contexts, thereby reducing memory access and latency. Implemented in SGLang for Qwen3.5 and Nemotron‑3, KVBuffer achieves up to 1.47× faster batch decoding and a five‑fold increase in speculative decoding throughput when verifying four draft tokens.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv Machine Learning

Aug 18

DeltaLog: Deferred Materialization of Recurrent States for Linear Attention Decoding

arXiv:2608. 15533v1 Announce Type: cross Abstract: Linear attention models eliminate the quadratic prefix computation and context-growing KV cache of softmax attention by replacing pairwise token interactions with recurrent state updates.

By Junqing Lin, Jingwei Sun, Guangzhong Sun

efficiency

arXiv Machine Learning

Jun 2

Vegas: Self-Speculative Decoding with Verification-Guided Sparse Attention

arXiv:2602. 07223v2 Announce Type: replace Abstract: Long-context large language model (LLM) inference has become the norm for today's AI applications.

By Yikang Yue, Yuqi Xue, Jian Huang

llms efficiency benchmarks

arXiv AI

Oct 1

ResidualKV: Residual-Based KV Cache Compression for Efficient Long-Context Inference

arXiv:2602.08005v2 Announce Type: replace-cross Abstract: Efficient long-context inference faces two coupled bottlenecks: KV-cache memory grows linearly with context length, while attention computati...

By Jitai Hao, Qiang Huang, Yaowei Wang, Min Zhang, Jun Yu

llms efficiency multimodal

arXiv Machine Learning

Jul 3

Lynx: Progressive Speculative Quantization for accelerating KV Transfer in Long-Context Inference

arXiv:2607. 01831v1 Announce Type: cross Abstract: Long-context inference is increasingly common in large language model (LLM) serving, driven by retrieval-augmented generation and agentic systems.

By Wenchen Han, Gingfung Matthew Yeung, Marco Barletta, William Toner, Amory Hoste, Adam Barker

llms rag agents efficiency benchmarks

arXiv Machine Learning

Jun 3

Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Serving

arXiv:2606. 02964v1 Announce Type: cross Abstract: Large Language Model (LLM) inference relies on key-value (KV) caches to avoid redundant attention computation.

By Chunan Shi, Yilei Chen, Yilin Chen, Xupeng Miao, Bin Cui

llms rag agents efficiency

arXiv AI

Sep 4

GrowPage: On-Demand KV Budgeting for Efficient LLM Reasoning Serving

GrowPage is an on‑demand key–value (KV) budgeting framework designed to improve large language model (LLM) reasoning serving. It treats KV capacity as a runtime resource, using lightweight dual‑timescale query summaries to track recent and long‑term attention patterns and estimate demand evolution. At each capacity boundary, GrowPage either compresses KV states within the current allocation or acquires an additional physical page, integrating with PagedAttention’s page‑level memory abstraction to maintain continuous batching and prefix caching.

By Qiankun Ma, Yanjiang Zhou, Zinan Xiong, Haofei Wang, Zhen Song, Yang Xiang, Ziyao Zhang, Hairong Zheng

llms efficiency benchmarks

One email a morning, machine-written

Related stories

DeltaLog: Deferred Materialization of Recurrent States for Linear Attention Decoding

Vegas: Self-Speculative Decoding with Verification-Guided Sparse Attention

ResidualKV: Residual-Based KV Cache Compression for Efficient Long-Context Inference

Lynx: Progressive Speculative Quantization for accelerating KV Transfer in Long-Context Inference

Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Serving

GrowPage: On-Demand KV Budgeting for Efficient LLM Reasoning Serving