arXiv Machine Learning
Jun 30
Multi-Block Diffusion Language Models
arXiv:2606. 29215v1 Announce Type: new Abstract: Block Diffusion Language Models (BD-LMs) improve diffusion-based text generation with KV caching and flexible-length generation.
Block Parallelism For Efficient Distributed Long-Context Diffusion Language Model Training
The paper introduces Block Parallelism (BP) and Context‑Sharded Block Parallelism (CSBP) to improve training efficiency for Block Diffusion Language Models (BDLMs) with long contexts. By assigning each corrupted‑block computation to a separate rank and sharding the shared clean sequence, CSBP reduces communication overhead and memory usage while preserving training semantics. Experiments on 16 H200 GPUs and 8 H100 GPUs show throughput gains of up to 1.61× and 7.59×, respectively, and higher benchmark pass rates in practical fine‑tuning scenarios.
Related stories
Hugging Face Trending Papers
Jul 20
FlowBlock: Wavefront-Parallel Decoding for Self-Correcting Diffusion Language Models
Block-wise diffusion large language models (dLLMs) decode sequentially at the block level, enabling effective KV-cache reuse across blocks but making inter-block decoding strictly serial. Prior work has attempted to unlock inter-block parallelism through post-training methods, but achieves only modest speedups and often degrades accuracy.
arXiv AI
Jul 21
FlowBlock: Wavefront-Parallel Decoding for Self-Correcting Diffusion Language Models
arXiv:2607. 17652v1 Announce Type: new Abstract: Block-wise diffusion large language models (dLLMs) decode sequentially at the block level, enabling effective KV-cache reuse across blocks but making inter-block decoding strictly serial.
arXiv AI
Jun 2
BlockBatch: Multi-Scale Consensus Decoding for Efficient Diffusion Language Model Inference
arXiv:2605. 29233v2 Announce Type: replace-cross Abstract: Diffusion language models (dLLMs) generate text by iteratively denoising multiple token positions in parallel, offering an attractive alternative to strictly autoregressive decoding.
arXiv AI
Jul 7
Training Hybrid Block Diffusion Language Models with Partial Bidirectionality
arXiv:2607. 02805v1 Announce Type: cross Abstract: High-throughput long-context generation is one of the central challenges for large language models.
arXiv Machine Learning
Aug 10
Retrofitting Linear Attention into Diffusion Language Models
arXiv:2608. 06628v1 Announce Type: new Abstract: Diffusion language models (dLLMs) offer a promising alternative to autoregressive models by accelerating inference through parallel decoding.
arXiv Machine Learning
Sep 14
Fixed State, Long Reach: What a Constant-Size Cache Buys Block Diffusion at Scale
The paper investigates how block‑diffusion language models can use a constant‑size cache to enable efficient parallel decoding. By employing sequence mixers that summarize completed blocks into a reusable state and a block‑causal training objective, the authors pretrain three 3B block‑diffusion denoisers (attention, Mamba, and hybrid) on 300 B tokens. The resulting state‑space cache remains O(1) in memory and latency regardless of context length, yielding significant speed‑up and memory savings compared to traditional attention‑based caches, especially at very long sequences.
arXiv AI
Jul 28
Beyond Block Boundaries: Multi-Block Editing for Diffusion Large Language Models
arXiv:2607. 22663v1 Announce Type: new Abstract: Block diffusion has emerged as the dominant paradigm for scaling discrete diffusion language models (dLLMs), because decoding text in fixed-size blocks preserves parallel generation within each block while keeping the quadratic attention cost tractable.
arXiv Machine Learning
Jun 8
MAGE: All-[MASK] Block Already Knows Where to Look in Block Diffusion LLM
arXiv:2602. 14209v2 Announce Type: replace Abstract: Block diffusion LLMs are an emerging paradigm for parallel language generation, but their KV caching makes memory access the dominant bottleneck in long-context inference.
arXiv AI
Jun 12
MiniMax Sparse Attention
arXiv:2606. 13392v1 Announce Type: new Abstract: Ultra-long-context capability is becoming indispensable for frontier LLMs: agentic workflows, repository-scale code reasoning, and persistent memory all require the model to jointly attend over hundreds of thousands to millions of tokens, yet the quadratic cost of softmax attention makes this untenable at deployment scale.
arXiv Machine Learning
Jul 3
SCAPE: Accurate and Efficient LLM Training with Extreme Sparse Communication
arXiv:2607. 01678v1 Announce Type: new Abstract: Communication increasingly dominates the cost of Large Language Model (LLM) pre-training, especially under data-parallel and sharded training schemes, where gradient synchronization and parameter reconstruction overhead increase with model size and system scale.
arXiv AI
Jun 19
LoRDO: Distributed Low-Rank Optimization with Infrequent Communication
arXiv:2602. 04396v2 Announce Type: replace-cross Abstract: Distributed training of foundation models via $\texttt{DDP}$ is limited by interconnect bandwidth.