The paper introduces Sigma, a large-scale continuous diffusion language model (3B/8B parameters) that uses steerable, low-dimensional ODE/SDE latent trajectories to address the non-smoothness of discrete diffusion models. Sigma is trained blockwise via likelihood optimization, jointly denoises Gaussian-corrupted token embeddings, and learns an optimal embedding geometry, leveraging pre-trained autoregressive weights for faster training. During inference, classifier-free guidance and score temperature are identified as essential for high-fidelity reasoning and coding, and Sigma matches or exceeds discrete models on benchmarks such as GSM8K, Minerva, HumanEval, MBPP, MATH-500, and AIME, while also revealing unique structural benefits like embedding-space steering and graceful degradation for low NFEs.
By Zhihan Yang, Wei Guo, Jean-Marie Lemercier, Simon Welker, Yonggan Fu, Mohammad Mahdi Kamani, Sajad Norouzi, Julius Berner, Tomas Geffner, Karsten Kreis, Yongxin Chen, Molei Tao, John Thickstun, Pavlo Molchanov, Ante Juki\'c, Arash Vahdat, Morteza Mardani
arXiv:2610.17590v1 Announce Type: cross
Abstract: Diffusion language models (DLMs) have attracted wide attention for their fast, parallel decoding, but they are believed to be far less efficient to t...
By Guanghan Wang, Bing He, Zhan Shi, Benoit Dumoulin, Hanqing Lu, Volodymyr Kuleshov
arXiv:2605. 13026v2 Announce Type: replace-cross Abstract: Masked diffusion models (MDMs) have emerged as a promising alternative to autoregressive models (ARMs) for language modeling.
By Chunsan Hong, Sanghyun Lee, Chieh-Hsin Lai, Satoshi Hayakawa, Yuhta Takida, Yuki Mitsufuji, Seungryong Kim, Jong Chul Ye
arXiv:2606. 15327v1 Announce Type: new Abstract: Diffusion Language Models (DLMs) have demonstrated strong scaling capacity as alternatives to autoregressive language models.
By Keyue Jiang, Yuxiang Wang, Yanan Zhao, Xiang Yu, Qifang Zhao, Bohan Tang, Baojian Zhou, Yanghua Xiao, Lin Qu, Xiaoxiao Xu
arXiv:2606. 19475v1 Announce Type: new Abstract: Large Language Models (LLMs) have revolutionized language modeling through autoregressive generation, enabling strong performance across a wide range of tasks.
By Thomas Bertolani, Davide Bucciarelli, Leonardo Zini, Marcella Cornia, Lorenzo Baraldi
arXiv:2605. 00161v2 Announce Type: replace Abstract: Diffusion language models (DLMs) are an attractive alternative to autoregressive models because they promise sublinear-time, parallel generation, yet practical gains remain elusive as high-quality samples still demand hundreds of refinement steps.
By Hasan Amin, Yuan Gao, Yaser Souri, Subhojit Som, Ming Yin, Rajiv Khanna, Xia Song
arXiv:2602. 19066v2 Announce Type: replace-cross Abstract: Diffusion Language Models (DLMs) have recently achieved strong results in text generation.
By David Li, Nikita Gushchin, Dmitry Abulkhanov, Eric Moulines, Ivan Oseledets, Maxim Panov, Alexander Korotin
arXiv:2606. 19005v1 Announce Type: cross Abstract: Diffusion models have become a promising alternative to autoregressive models.
By Mengyu Ye, Keito Kudo, Wataru Ikeda, Ryosuke Matsuda, Keisuke Sakaguchi, Jun Suzuki
The paper examines whether exponential moving average (EMA) is uniquely suited for training large diffusion models or if other optimization strategies can be more effective. It shows that constant‑step averaging has inherent limitations, while learning‑rate decay and Schedule‑Free optimization provide stronger convergence guarantees under nonconvex assumptions. Empirical results indicate that combining decay with EMA yields modest gains, but Schedule‑Free consistently outperforms EMA alone, reducing parameter variability while maintaining late‑stage progress and improving generation quality across multiple diffusion settings.
By Chuyu Hu, Huanran Chen, Shitong Shao, Yinpeng Dong
arXiv:2508. 10875v3 Announce Type: replace-cross Abstract: Diffusion Language Models (DLMs) are rapidly emerging as a powerful and promising alternative to the dominant autoregressive (AR) paradigm.
By Tianyi Li, Mingda Chen, Bowei Guo, Zhiqiang Shen
The article "How To Build A Modern Diffusion Language Model" reviews the recent advances in large language models (LLMs) and highlights their limitations, such as sequential token generation that underutilizes GPU hardware and prevents error correction. It explains how discrete diffusion models overcome these issues by generating tokens in parallel through iterative denoising, enabling high-speed inference and controllable generation. The paper surveys key concepts, architectural components, and applications of diffusion LLMs in biology and language, and discusses their implications for future scaling laws.
By Volodymyr Kuleshov, Marianne Arriola, Yair Schiff, Gilad Turok, Guanghan Wang
The paper introduces Less Uniform Diffusion (LUDI), a framework that improves uniform diffusion language models by using a less uniform loss and per-token time embeddings to guide reverse transitions and enable confidence-based few-step sampling. Experiments demonstrate that LUDI provides cleaner supervision, enhances few-step generation, and scales to a 7B model (LUDI-7B) that achieves a 3-token-per-step speedup over autoregressive decoding while matching masked diffusion baselines. The work suggests that UDLMs still have untapped potential for complex generation tasks.
By Kaibo Wang, Ding Ding, Fangyu Ding, Zijin Feng, Han Shi, Haili Bai, Jiacheng Sun, Yang Xiang