arXiv Machine Learning

Decode Token Parallelism: Memory-Efficient Distributed Attention Parallelism for Scalable LLM Decoding

The paper introduces Decode Token Parallelism (TKNP), a new decoding architecture that separates model weight placement from attention computation. TKNP distributes a single model copy across nodes in balanced layer blocks, allowing each node to compute attention for its own requests using local KV caches while performing dense operations within a single NVLink domain. Implemented in vLLM, TKNP achieves up to 1.7× higher throughput than decode context parallelism and up to 2.06× over data parallelism on large models such as Llama 3.1 405B and Nemotron‑4 340B across 64 H100 GPUs.

arXiv Machine Learning
Sep 14

Fixed State, Long Reach: What a Constant-Size Cache Buys Block Diffusion at Scale

The paper investigates how block‑diffusion language models can use a constant‑size cache to enable efficient parallel decoding. By employing sequence mixers that summarize completed blocks into a reusable state and a block‑causal training objective, the authors pretrain three 3B block‑diffusion denoisers (attention, Mamba, and hybrid) on 300 B tokens. The resulting state‑space cache remains O(1) in memory and latency regardless of context length, yielding significant speed‑up and memory savings compared to traditional attention‑based caches, especially at very long sequences.

By Vaibhav Singh, Pierre-Andr\'e No\"el, Torsten Scholak, Eugene Belilovsky, Oleksiy Ostapenko
arXiv Machine Learning
Oct 15

GQLA: Group-Query Latent Attention for Hardware-Adaptive Decoding

The paper introduces Group-Query Latent Attention (GQLA), a variant of Multi-head Latent Attention that shares group‑sharded up‑projections to enable a joint latent representation and a shardable expanded cache. GQLA supports both MQA‑absorb and GQA execution using the same weights and existing kernels, and roofline analysis shows how absorption trades traffic for computation while GQA shards KV groups. Experiments on DeepSeek‑V3.1‑Base and other models demonstrate that GQLA achieves comparable training losses to MLA, maintains high accuracy after checkpoint conversion, and improves output throughput by up to 35% on eight H20‑3e GPUs, while also allowing larger batch sizes due to a smaller per‑rank cache.

By Fanxu Meng, Zhixin Pan
arXiv Machine Learning
Oct 9

QUILT: Rethinking Sparse-Attention Prefill through Shared Query Execution

The paper introduces QUILT, a new sparse‑attention execution mechanism that jointly processes neighboring queries to reuse shared key‑value (KV) entries, thereby reducing redundant memory traffic and computation. QUILT employs Shift‑and‑Compare Set Decomposition (SCSD) to convert irregular set operations into regular data‑parallel primitives and pipelines this with attention computation to hide overhead. The approach includes cascaded sharing across multiple granularities and a tile‑aware strategy that balances sharing with hardware utilization, achieving up to 55.1% lower kernel latency and 36.8% faster time‑to‑first‑token on LongBench benchmarks with negligible accuracy loss.

By Zhenduo Zhao, Qihui Zhou, Mingcong Song, Zhiyi Chen, Chuangguan Ye, Fengfan Hou, Zequn Gong, Jing Li, Hongjie Si, Guoping Long
arXiv AI
Nov 6

KVBuffer: IO-aware Serving for Linear Attention

KVBuffer is an IO‑aware serving mechanism designed to improve the efficiency of linear attention models. By buffering recent keys and values, it enables chunkwise decoding, parallel speculative decoding, and direct attention computation for short contexts, thereby reducing memory access and latency. Implemented in SGLang for Qwen3.5 and Nemotron‑3, KVBuffer achieves up to 1.47× faster batch decoding and a five‑fold increase in speculative decoding throughput when verifying four draft tokens.

By Longwei Zou, Lin Zhong
arXiv Machine Learning
Aug 4

Bole: Efficient Tree Speculation for Hybrid-Attention Language Models

arXiv:2608. 01651v1 Announce Type: cross Abstract: Hybrid-attention large language models combine full attention with recurrent linear attention to reduce long-context inference costs, yet their autoregressive decoding remains memory-bound.

By Li Wang, Yi Su, Xiabao Wu, Chiran You, Yongchao Liu, Zhan Qiu, Juelu Zhang, Jiajun Zheng, Fangxin Liu, Jie Zhang, Chen Tian, Chengying Huan
arXiv AI
Nov 5

Split-KV: Efficient Context Parallelism for Training Diffusion Drafters

Split-KV introduces a context‑parallelism technique for training diffusion drafters, keeping key/value shards local and limiting per‑layer attention communication to query‑sized tensors. This approach exploits the asymmetry between short draft queries and long contexts, enabling efficient load balancing via a cost‑aware longest‑processing‑time‑first strategy. Experiments show that at a CP size of 64, Split‑KV achieves 85–91 % weak‑scaling efficiency and up to 3.67× speedup over the best baseline, with less than 1 % load imbalance.

By Geng Zhang, Fangzhao Zhang, Cong Xie, Tiantian Fan, Yang You
arXiv Machine Learning
Sep 17

GroupKV: Hierarchical KV Cache Management for Long-Context Diffusion LLM Inference

GroupKV is a lightweight hierarchical KV cache management system designed for long‑context diffusion large language model (dLLM) inference. It partitions the context into contiguous groups and uses coarse‑to‑fine sparse selection, cross‑layer consistency for predictive prefetching, and a staleness correction mechanism to keep the cache coherent amid dynamic KV updates. The approach also incorporates streaming prefill to lower peak memory usage, achieving up to 48× longer serviceable context, 3.73× faster inference in offload‑based settings, and competitive task accuracy.

By Jinhao Wang, Zhexin Hu, Kangjie Zhou, Xin Zhou, Fangfang Liu
arXiv Machine Learning
Sep 18

Block Parallelism For Efficient Distributed Long-Context Diffusion Language Model Training

The paper introduces Block Parallelism (BP) and Context‑Sharded Block Parallelism (CSBP) to improve training efficiency for Block Diffusion Language Models (BDLMs) with long contexts. By assigning each corrupted‑block computation to a separate rank and sharding the shared clean sequence, CSBP reduces communication overhead and memory usage while preserving training semantics. Experiments on 16 H200 GPUs and 8 H100 GPUs show throughput gains of up to 1.61× and 7.59×, respectively, and higher benchmark pass rates in practical fine‑tuning scenarios.

By Tarun Suresh, Pranshu Chaturvedi, Hangoo Kang, Parth Shroff, Ishan S. Khare, Hermann Kumbong, Azalia Mirhoseini