arXiv Machine Learning

Parallelism, critical windows, and separations among diffusion language models

The paper compares the parallelism capabilities of three diffusion large language model paradigms—masked, uniform, and Gaussian diffusion. It proves that uniform and Gaussian diffusion can sample with a number of forward passes scaling with the dual total correlation of the distribution, potentially much less than the context length, whereas masked diffusion may require more passes. The study establishes a provable separation in parallelism, showing that masked diffusion’s critical windows are asymptotically narrower than those of the other two approaches.

arXiv Computation and Language
Oct 22

Hierarchical Masked Diffusion Language Models

arXiv:2610.29255v1 Announce Type: new Abstract: Masked diffusion language models have emerged as an alternative to autoregressive models, enabling parallel token generation. However, standard masked...

By Parul Gupta, Satwik, Avni Aggarwal, Bhavesh Gurnani, Arnab Kumar Mondal, Dinesh Khandelwal, Dinesh Garg, Parag Singla
arXiv AI
Jun 10

Attention-Discounted Adaptive Sampler for Masked Diffusion Language Models

arXiv:2606. 10829v1 Announce Type: cross Abstract: Masked diffusion language models can reduce inference steps by revealing multiple tokens per denoising iteration, but this parallelism is fragile: positions that are individually confident may be unsafe to commit together when their predictions are coupled.

By Yusuf Sahin, Ahmed Rockey Saikia, Volkan Cevher, Paolo Favaro
arXiv Machine Learning
Sep 11

Continuous Diffusion Scales Competitively with Discrete Diffusion for Language

The paper revisits the continuous diffusion language model Plaid and introduces RePlaid, aligning its architecture with modern discrete diffusion models. RePlaid achieves a compute gap of only 20× compared to autoregressive models, surpasses Duo with fewer parameters, and outperforms MDLM in over‑trained settings. On OpenWebText, RePlaid sets a new state‑of‑the‑art continuous diffusion perplexity of 22.1 and demonstrates superior generation quality, while theoretical analysis links likelihood‑based training to linear cross‑entropy over time and structured embedding geometries.

By Zhihan Yang, Wei Guo, Shuibai Zhang, Subham Sekhar Sahoo, Yongxin Chen, Arash Vahdat, Morteza Mardani, John Thickstun
arXiv AI
Nov 6

Masked diffusion as random sequential adsorption: an exactly solvable model for parallel decoding

Masked diffusion models generate text by reversing a stochastic masking process, allowing many tokens to be revealed in parallel at the cost of errors due to independent token draws. The paper maps this trade‑off onto driven many‑particle systems, showing that for Markov data the process is a solvable random sequential adsorption model and that the error equals entropy production relative to the reversed masking. For many steps, the error corresponds to dissipation in a slowly driven system, with optimal schedules being geodesics of a thermodynamic metric derived from token mutual information, revealing how text correlations influence error concentration and output quality.

By Adrian Baule