arXiv Machine Learning September 18, 2026 By Tarun Suresh, Pranshu Chaturvedi, Hangoo Kang, Parth Shroff, Ishan S. Khare, Hermann Kumbong, Azalia Mirhoseini

Block Parallelism For Efficient Distributed Long-Context Diffusion Language Model Training

Read the original on arXiv Machine Learning →

The paper introduces Block Parallelism (BP) and Context‑Sharded Block Parallelism (CSBP) to improve training efficiency for Block Diffusion Language Models (BDLMs) with long contexts. By assigning each corrupted‑block computation to a separate rank and sharding the shared clean sequence, CSBP reduces communication overhead and memory usage while preserving training semantics. Experiments on 16 H200 GPUs and 8 H100 GPUs show throughput gains of up to 1.61× and 7.59×, respectively, and higher benchmark pass rates in practical fine‑tuning scenarios.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Machine Learning

Jun 30

Multi-Block Diffusion Language Models

arXiv:2606. 29215v1 Announce Type: new Abstract: Block Diffusion Language Models (BD-LMs) improve diffusion-based text generation with KV caching and flexible-length generation.

By Yijie Jin, Jiajun Xu, Yuxuan Liu, Chenkai Xu, Yi Tu, Jiajun Li, Dandan Tu, Xiaohui Yan, Kai Yu, Pengfei Liu, Zhijie Deng

llms diffusion benchmarks

Hugging Face Trending Papers

Jul 20

FlowBlock: Wavefront-Parallel Decoding for Self-Correcting Diffusion Language Models

Block-wise diffusion large language models (dLLMs) decode sequentially at the block level, enabling effective KV-cache reuse across blocks but making inter-block decoding strictly serial. Prior work has attempted to unlock inter-block parallelism through post-training methods, but achieves only modest speedups and often degrades accuracy.

llms diffusion efficiency benchmarks

arXiv AI

Jul 21

FlowBlock: Wavefront-Parallel Decoding for Self-Correcting Diffusion Language Models

arXiv:2607. 17652v1 Announce Type: new Abstract: Block-wise diffusion large language models (dLLMs) decode sequentially at the block level, enabling effective KV-cache reuse across blocks but making inter-block decoding strictly serial.

By Bing Tian, Haikun Liu, Xiaocheng Zhong, Zhuohui Duan, Zhaokai Luo, Huayi Jin, Zhiyong Wang, Xiaofei Liao

llms diffusion efficiency benchmarks

arXiv AI

Jun 2

BlockBatch: Multi-Scale Consensus Decoding for Efficient Diffusion Language Model Inference

arXiv:2605. 29233v2 Announce Type: replace-cross Abstract: Diffusion language models (dLLMs) generate text by iteratively denoising multiple token positions in parallel, offering an attractive alternative to strictly autoregressive decoding.

By Xiaoyou Wu, Cheng-Jhih Shih, Binfei Ji, Yong Liu, Yingyan Celine Lin

llms diffusion

arXiv AI

Jul 7

Training Hybrid Block Diffusion Language Models with Partial Bidirectionality

arXiv:2607. 02805v1 Announce Type: cross Abstract: High-throughput long-context generation is one of the central challenges for large language models.

By Pranshu Chaturvedi, Parth Shroff, Tarun Suresh, Hangoo Kang, Kaiyue Wen

llms diffusion efficiency

arXiv Machine Learning

Aug 10

Retrofitting Linear Attention into Diffusion Language Models

arXiv:2608. 06628v1 Announce Type: new Abstract: Diffusion language models (dLLMs) offer a promising alternative to autoregressive models by accelerating inference through parallel decoding.

By Jinha Kim, Younghun Roh, Jaeyeon Kim

llms diffusion efficiency benchmarks

One email a morning, machine-written

Related stories

Multi-Block Diffusion Language Models

FlowBlock: Wavefront-Parallel Decoding for Self-Correcting Diffusion Language Models

FlowBlock: Wavefront-Parallel Decoding for Self-Correcting Diffusion Language Models

BlockBatch: Multi-Scale Consensus Decoding for Efficient Diffusion Language Model Inference

Training Hybrid Block Diffusion Language Models with Partial Bidirectionality

Retrofitting Linear Attention into Diffusion Language Models