arXiv Machine Learning By Susav Shrestha, Brad Settlemyer, Narasimha Reddy

Decode Token Parallelism: Memory-Efficient Distributed Attention Parallelism for Scalable LLM Decoding

Read the original on arXiv Machine Learning →

The paper introduces Decode Token Parallelism (TKNP), a new decoding architecture that separates model weight placement from attention computation. TKNP distributes a single model copy across nodes in balanced layer blocks, allowing each node to compute attention for its own requests using local KV caches while performing dense operations within a single NVLink domain. Implemented in vLLM, TKNP achieves up to 1.7× higher throughput than decode context parallelism and up to 2.06× over data parallelism on large models such as Llama 3.1 405B and Nemotron‑4 340B across 64 H100 GPUs.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Machine Learning
Sep 14

Fixed State, Long Reach: What a Constant-Size Cache Buys Block Diffusion at Scale

The paper investigates how block‑diffusion language models can use a constant‑size cache to enable efficient parallel decoding. By employing sequence mixers that summarize completed blocks into a reusable state and a block‑causal training objective, the authors pretrain three 3B block‑diffusion denoisers (attention, Mamba, and hybrid) on 300 B tokens. The resulting state‑space cache remains O(1) in memory and latency regardless of context length, yielding significant speed‑up and memory savings compared to traditional attention‑based caches, especially at very long sequences.

By Vaibhav Singh, Pierre-Andr\'e No\"el, Torsten Scholak, Eugene Belilovsky, Oleksiy Ostapenko
arXiv Machine Learning
Oct 15

GQLA: Group-Query Latent Attention for Hardware-Adaptive Decoding

The paper introduces Group-Query Latent Attention (GQLA), a variant of Multi-head Latent Attention that shares group‑sharded up‑projections to enable a joint latent representation and a shardable expanded cache. GQLA supports both MQA‑absorb and GQA execution using the same weights and existing kernels, and roofline analysis shows how absorption trades traffic for computation while GQA shards KV groups. Experiments on DeepSeek‑V3.1‑Base and other models demonstrate that GQLA achieves comparable training losses to MLA, maintains high accuracy after checkpoint conversion, and improves output throughput by up to 35% on eight H20‑3e GPUs, while also allowing larger batch sizes due to a smaller per‑rank cache.

By Fanxu Meng, Zhixin Pan
arXiv Machine Learning
Oct 9

QUILT: Rethinking Sparse-Attention Prefill through Shared Query Execution

The paper introduces QUILT, a new sparse‑attention execution mechanism that jointly processes neighboring queries to reuse shared key‑value (KV) entries, thereby reducing redundant memory traffic and computation. QUILT employs Shift‑and‑Compare Set Decomposition (SCSD) to convert irregular set operations into regular data‑parallel primitives and pipelines this with attention computation to hide overhead. The approach includes cascaded sharing across multiple granularities and a tile‑aware strategy that balances sharing with hardware utilization, achieving up to 55.1% lower kernel latency and 36.8% faster time‑to‑first‑token on LongBench benchmarks with negligible accuracy loss.

By Zhenduo Zhao, Qihui Zhou, Mingcong Song, Zhiyi Chen, Chuangguan Ye, Fengfan Hou, Zequn Gong, Jing Li, Hongjie Si, Guoping Long
arXiv AI
Nov 6

KVBuffer: IO-aware Serving for Linear Attention

KVBuffer is an IO‑aware serving mechanism designed to improve the efficiency of linear attention models. By buffering recent keys and values, it enables chunkwise decoding, parallel speculative decoding, and direct attention computation for short contexts, thereby reducing memory access and latency. Implemented in SGLang for Qwen3.5 and Nemotron‑3, KVBuffer achieves up to 1.47× faster batch decoding and a five‑fold increase in speculative decoding throughput when verifying four draft tokens.

By Longwei Zou, Lin Zhong