arXiv Machine Learning

Block Parallelism For Efficient Distributed Long-Context Diffusion Language Model Training

The paper introduces Block Parallelism (BP) and Context‑Sharded Block Parallelism (CSBP) to improve training efficiency for Block Diffusion Language Models (BDLMs) with long contexts. By assigning each corrupted‑block computation to a separate rank and sharding the shared clean sequence, CSBP reduces communication overhead and memory usage while preserving training semantics. Experiments on 16 H200 GPUs and 8 H100 GPUs show throughput gains of up to 1.61× and 7.59×, respectively, and higher benchmark pass rates in practical fine‑tuning scenarios.

arXiv Machine Learning
Jun 30

Multi-Block Diffusion Language Models

arXiv:2606. 29215v1 Announce Type: new Abstract: Block Diffusion Language Models (BD-LMs) improve diffusion-based text generation with KV caching and flexible-length generation.

By Yijie Jin, Jiajun Xu, Yuxuan Liu, Chenkai Xu, Yi Tu, Jiajun Li, Dandan Tu, Xiaohui Yan, Kai Yu, Pengfei Liu, Zhijie Deng
Hugging Face Trending Papers
Jul 20

FlowBlock: Wavefront-Parallel Decoding for Self-Correcting Diffusion Language Models

Block-wise diffusion large language models (dLLMs) decode sequentially at the block level, enabling effective KV-cache reuse across blocks but making inter-block decoding strictly serial. Prior work has attempted to unlock inter-block parallelism through post-training methods, but achieves only modest speedups and often degrades accuracy.

arXiv Machine Learning
6d ago

Decode Token Parallelism: Memory-Efficient Distributed Attention Parallelism for Scalable LLM Decoding

The paper introduces Decode Token Parallelism (TKNP), a new decoding architecture that separates model weight placement from attention computation. TKNP distributes a single model copy across nodes in balanced layer blocks, allowing each node to compute attention for its own requests using local KV caches while performing dense operations within a single NVLink domain. Implemented in vLLM, TKNP achieves up to 1.7× higher throughput than decode context parallelism and up to 2.06× over data parallelism on large models such as Llama 3.1 405B and Nemotron‑4 340B across 64 H100 GPUs.

By Susav Shrestha, Brad Settlemyer, Narasimha Reddy
arXiv Machine Learning
Sep 14

Fixed State, Long Reach: What a Constant-Size Cache Buys Block Diffusion at Scale

The paper investigates how block‑diffusion language models can use a constant‑size cache to enable efficient parallel decoding. By employing sequence mixers that summarize completed blocks into a reusable state and a block‑causal training objective, the authors pretrain three 3B block‑diffusion denoisers (attention, Mamba, and hybrid) on 300 B tokens. The resulting state‑space cache remains O(1) in memory and latency regardless of context length, yielding significant speed‑up and memory savings compared to traditional attention‑based caches, especially at very long sequences.

By Vaibhav Singh, Pierre-Andr\'e No\"el, Torsten Scholak, Eugene Belilovsky, Oleksiy Ostapenko
Hugging Face Trending Papers
Oct 13

Prefilling-dLLM v2: Learned Chunk Routing for Long-Context Diffusion Language Models

Prefilling-dLLM v2 is a diffusion language model that learns to route computation using a native mask token, enabling efficient sparse attention across long contexts. By appending a mask at the end of each chunk and using a learned role offset, the model selectively attends to top‑K chunks every fourth layer while maintaining local windows elsewhere. Evaluated on 16K–128K contexts, Prefilling-dLLM v2 outperforms other sparse methods (NSA, DSA, SWA) on RULER tasks and achieves a 172× speedup over full dense attention at 128K tokens.

arXiv AI
6d ago

Blurring the Block Boundaries: Overlapping Block Denoising for Efficient Diffusion Language Models

The paper introduces Blurring the Block Boundaries (B3), a technique that allows two blocks in a diffusion language model to be refined simultaneously during a single forward pass. B3 dynamically opens the next block only when the model is ready, eliminating the need for hyperparameters or additional training. Experiments on ten datasets show that B3 maintains generation quality while achieving an 11.5–14.5% speedup over baseline models.

By Amir Mann, Jonathan Fhima, Shaked Brody, Or Avitan, Asif Zamir, Shahar Tsiper, Niv Nayman
arXiv AI
Jul 28

Beyond Block Boundaries: Multi-Block Editing for Diffusion Large Language Models

arXiv:2607. 22663v1 Announce Type: new Abstract: Block diffusion has emerged as the dominant paradigm for scaling discrete diffusion language models (dLLMs), because decoding text in fixed-size blocks preserves parallel generation within each block while keeping the quadratic attention cost tractable.

By Xingyu Mou, Zijin Huang, Tianze Zhang, Yuxin Ma, Lanning Wei, Zengfeng Huang, Da Zheng, Lun Du