The paper investigates how block‑diffusion language models can use a constant‑size cache to enable efficient parallel decoding. By employing sequence mixers that summarize completed blocks into a reusable state and a block‑causal training objective, the authors pretrain three 3B block‑diffusion denoisers (attention, Mamba, and hybrid) on 300 B tokens. The resulting state‑space cache remains O(1) in memory and latency regardless of context length, yielding significant speed‑up and memory savings compared to traditional attention‑based caches, especially at very long sequences.
By Vaibhav Singh, Pierre-Andr\'e No\"el, Torsten Scholak, Eugene Belilovsky, Oleksiy Ostapenko
The paper introduces Group-Query Latent Attention (GQLA), a variant of Multi-head Latent Attention that shares group‑sharded up‑projections to enable a joint latent representation and a shardable expanded cache. GQLA supports both MQA‑absorb and GQA execution using the same weights and existing kernels, and roofline analysis shows how absorption trades traffic for computation while GQA shards KV groups. Experiments on DeepSeek‑V3.1‑Base and other models demonstrate that GQLA achieves comparable training losses to MLA, maintains high accuracy after checkpoint conversion, and improves output throughput by up to 35% on eight H20‑3e GPUs, while also allowing larger batch sizes due to a smaller per‑rank cache.
By Fanxu Meng, Zhixin Pan
arXiv:2606. 06256v1 Announce Type: new Abstract: As the input length of large language model (LLM) serving continues to grow, the KV cache has become a dominant bottleneck in AI infrastructure.
By Yang Liu, ZhaoKai Luo, HuaYi Jin, ZhiYong Wang, RuoZhou He, BoYu Wang, Guanjie Chen, Junhao Hu
arXiv:2609.37626v1 Announce Type: cross
Abstract: No single way of parallelizing attention serves large language models well under all loads. Low concurrency favors tensor parallelism, many independe...
By Chuan Liu, Shuoming Zhang, Zhicheng Li, Qianqi Sun, Ruiyuan Xu, Qiuchu Yu, Xiyu Shi, Huimin Cui, Jiacheng Zhao
The paper introduces QUILT, a new sparse‑attention execution mechanism that jointly processes neighboring queries to reuse shared key‑value (KV) entries, thereby reducing redundant memory traffic and computation. QUILT employs Shift‑and‑Compare Set Decomposition (SCSD) to convert irregular set operations into regular data‑parallel primitives and pipelines this with attention computation to hide overhead. The approach includes cascaded sharing across multiple granularities and a tile‑aware strategy that balances sharing with hardware utilization, achieving up to 55.1% lower kernel latency and 36.8% faster time‑to‑first‑token on LongBench benchmarks with negligible accuracy loss.
By Zhenduo Zhao, Qihui Zhou, Mingcong Song, Zhiyi Chen, Chuangguan Ye, Fengfan Hou, Zequn Gong, Jing Li, Hongjie Si, Guoping Long
KVBuffer is an IO‑aware serving mechanism designed to improve the efficiency of linear attention models. By buffering recent keys and values, it enables chunkwise decoding, parallel speculative decoding, and direct attention computation for short contexts, thereby reducing memory access and latency. Implemented in SGLang for Qwen3.5 and Nemotron‑3, KVBuffer achieves up to 1.47× faster batch decoding and a five‑fold increase in speculative decoding throughput when verifying four draft tokens.
By Longwei Zou, Lin Zhong
arXiv:2608. 01651v1 Announce Type: cross Abstract: Hybrid-attention large language models combine full attention with recurrent linear attention to reduce long-context inference costs, yet their autoregressive decoding remains memory-bound.
By Li Wang, Yi Su, Xiabao Wu, Chiran You, Yongchao Liu, Zhan Qiu, Juelu Zhang, Jiajun Zheng, Fangxin Liu, Jie Zhang, Chen Tian, Chengying Huan
arXiv:2606. 02964v1 Announce Type: cross Abstract: Large Language Model (LLM) inference relies on key-value (KV) caches to avoid redundant attention computation.
By Chunan Shi, Yilei Chen, Yilin Chen, Xupeng Miao, Bin Cui
Split-KV introduces a context‑parallelism technique for training diffusion drafters, keeping key/value shards local and limiting per‑layer attention communication to query‑sized tensors. This approach exploits the asymmetry between short draft queries and long contexts, enabling efficient load balancing via a cost‑aware longest‑processing‑time‑first strategy. Experiments show that at a CP size of 64, Split‑KV achieves 85–91 % weak‑scaling efficiency and up to 3.67× speedup over the best baseline, with less than 1 % load imbalance.
By Geng Zhang, Fangzhao Zhang, Cong Xie, Tiantian Fan, Yang You
arXiv:2609.24197v1 Announce Type: new
Abstract: Speculative decoding losslessly accelerates large language model inference by having a lightweight draft model predict future tokens for verification b...
By Weifan Jiang, Krishna Teja Chitty-Venkata, Megan Flynn, Reed Meyerson, Zhenting Qi, Tianyu Wu, Eldar Kurtic, Minlan Yu, Alexandre Marques
GroupKV is a lightweight hierarchical KV cache management system designed for long‑context diffusion large language model (dLLM) inference. It partitions the context into contiguous groups and uses coarse‑to‑fine sparse selection, cross‑layer consistency for predictive prefetching, and a staleness correction mechanism to keep the cache coherent amid dynamic KV updates. The approach also incorporates streaming prefill to lower peak memory usage, achieving up to 48× longer serviceable context, 3.73× faster inference in offload‑based settings, and competitive task accuracy.
By Jinhao Wang, Zhexin Hu, Kangjie Zhou, Xin Zhou, Fangfang Liu
The paper introduces Block Parallelism (BP) and Context‑Sharded Block Parallelism (CSBP) to improve training efficiency for Block Diffusion Language Models (BDLMs) with long contexts. By assigning each corrupted‑block computation to a separate rank and sharding the shared clean sequence, CSBP reduces communication overhead and memory usage while preserving training semantics. Experiments on 16 H200 GPUs and 8 H100 GPUs show throughput gains of up to 1.61× and 7.59×, respectively, and higher benchmark pass rates in practical fine‑tuning scenarios.
By Tarun Suresh, Pranshu Chaturvedi, Hangoo Kang, Parth Shroff, Ishan S. Khare, Hermann Kumbong, Azalia Mirhoseini