The paper introduces a lightweight single‑layer sampler that allows masked diffusion language models to approximate joint sampling of multiple tokens in a single full‑model forward pass. By training the sampler to mimic exact joint sampling from a frozen diffusion model, the authors enable parallel unmasking of tokens while maintaining a close match to the true joint distribution. Experiments on Dream‑7B and Llada‑7B models show that unmasking four tokens per denoising step yields a MAUVE score of 0.87, a substantial improvement over the marginal baseline of 0.31.
By Parikshit Bansal, Sujay Sanghavi
arXiv:2609.20830v2 Announce Type: replace
Abstract: Language models write text once, left to right, and models that write in other orders, such as diffusion LMs, pay for it with many full-sequence pa...
By Sean Diab
arXiv:2609.39560v1 Announce Type: new
Abstract: Sampling several responses and voting over their answers can improve a language model's accuracy, but repeated answers limit the benefit of additional...
By Michael Helcig, Martin Jaggi
The paper investigates neural text degeneration by measuring the fixed‑point structure of short‑window argmax maps across 17 pretrained models, using 96 random two‑token starts without prompts. It finds a stable four‑way classification that varies across model families and scales, with some models funneling to a single endpoint token while others do not, and shows that this behavior is not solely determined by training data or corpus frequency. The study demonstrates that repetition phenomena are not uniformly explained by either training data or network architecture alone, highlighting the complexity of neural text generation dynamics.
By Nicol\'as Vera Z\'u\~niga
Generative retrieval trains a language model to generate the identifier of a relevant document. Recent work replaces the autoregressive decoder with diffusion, but changes identifiers, training recipe...
arXiv:2606. 19005v1 Announce Type: cross Abstract: Diffusion models have become a promising alternative to autoregressive models.
By Mengyu Ye, Keito Kudo, Wataru Ikeda, Ryosuke Matsuda, Keisuke Sakaguchi, Jun Suzuki
arXiv:2508. 10875v3 Announce Type: replace-cross Abstract: Diffusion Language Models (DLMs) are rapidly emerging as a powerful and promising alternative to the dominant autoregressive (AR) paradigm.
By Tianyi Li, Mingda Chen, Bowei Guo, Zhiqiang Shen
arXiv:2606. 12232v1 Announce Type: new Abstract: Masked diffusion language models (dLLMs) have recently emerged as a competitive alternative to autoregressive language models, with the promise of faster inference via parallel token generation.
By Stipe Frkovic, Metod Jazbec, Dan Zhang, Christian A. Naesseth, Ilija Bogunovic, Eric Nalisnick
arXiv:2610.08716v1 Announce Type: cross
Abstract: Generative retrieval trains a language model to generate the identifier of a relevant document. Recent work replaces the autoregressive decoder with...
By Hicham Randrianarivo, Logan Renaud, Alexia Allal
arXiv:2605.27268v2 Announce Type: replace-cross
Abstract: Modern Large Language Models (LLMs) are often criticized for producing repetitive and homogeneous text, despite possessing vast latent vocabu...
By Samer Awad, Javier Conde, Carlos Arriaga, Tairan Fu, Javier Coronado-Bl\'azquez, Pedro Reviriego
arXiv:2610.14162v1 Announce Type: cross
Abstract: Diffusion language models (DLMs) offer parallel sampling for faster generation than autoregressive models, an increasingly important advantage as inf...
By Hermann Kumbong, Ishan S. Khare, Iddah Mlauzi, Andrew Woen, Charles Ding, E. Kelly Buchanan, Simran Arora, Azalia Mirhoseini, Christopher R\'e
Diffusion language models (dLLMs) have emerged as a promising alternative to autoregressive (AR) language models, offering flexible token-update orders and parallel decoding. Recent dLLMs are often in...