The paper introduces Representation-based Masked Diffusion Model (RMDM), a new framework for language modeling that improves upon existing Masked Diffusion Models by incorporating global semantic guidance. RMDM encodes text into a continuous semantic space with a pretrained encoder, normalizes this representation to a Gaussian prior via an invertible transformation, and then trains a masked diffusion model conditioned on this latent representation to coordinate parallel token updates. Experiments show that RMDM yields higher generation quality, especially when using aggressive few‑step sampling.
arXiv:2606. 04236v1 Announce Type: cross Abstract: Discrete diffusion language models can generate text efficiently by updating multiple masked positions in parallel, but this parallelism introduces a quality-latency trade-off.
arXiv:2508. 10875v3 Announce Type: replace-cross Abstract: Diffusion Language Models (DLMs) are rapidly emerging as a powerful and promising alternative to the dominant autoregressive (AR) paradigm.
The paper introduces Seq-Drifting, a new framework that enables diffusion language models to generate an entire text sequence in a single forward pass. By applying a drift field to continuous token representations, the model learns a direct mapping from Gaussian noise to text, using attraction to guide outputs toward contextually plausible tokens and repulsion to prevent repetition and collapse. Experiments show that Seq-Drifting achieves a perplexity of 36.01 on LM1B and performs competitively in unconditional, conditional, translation, summarization, and reasoning tasks.
arXiv:2606. 27617v1 Announce Type: cross Abstract: Masked Diffusion Models (MDMs) promise fast, parallel language generation, but their reverse transition factorises across token positions -- an approximation that breaks down in the few-step sampling regime where parallel generation ought to provide the greatest efficiency gains.
Diffusion large language models (DLLMs) enable non-autoregressive generation by iteratively denoising corrupted token sequences with bidirectional context. Despite their ability to update multiple positions in parallel, inference remains costly due to the many denoising steps required for high-quality generation.
arXiv:2601. 17917v3 Announce Type: replace Abstract: Diffusion Large Language Models (dLLMs) offer a compelling paradigm for natural language generation, leveraging parallel decoding and bidirectional attention to achieve superior global coherence compared to autoregressive models.
Cola2 is a hierarchical language modeling framework that jointly optimizes continuous semantic compression, reconstruction–generation balance, and continuous–discrete co‑adaptation. It dynamically segments text into semantic spans, controls compression via negative feedback, and trains a block‑causal masked diffusion decoder with clean, noisy, and dropped latent conditions. Experiments on five downstream tasks with 210 billion training tokens show competitive scaling, and an unconverged checkpoint with 346 billion source tokens outperforms Llama‑3‑8B on HellaSwag and StoryCloze.
AURORA-LM is a continuous‑latent diffusion language model that separates the creation of a high‑capacity, decodable text representation from the modeling of its distribution. It uses a Query‑based Encoder‑Decoder to produce a prefix‑aligned latent sequence and a Block‑Causal Diffusion Transformer to learn its distribution via flow matching, generating text block‑by‑block while denoising positions in parallel. The model calibrates noise levels to latent width, introduces self‑trajectory consistency, and achieves state‑of‑the‑art performance on OpenWebText generation and XSum summarization, outperforming larger publicly released latent‑diffusion models when scaled to 1 B parameters.
arXiv:2601.21647v2 Announce Type: replace-cross Abstract: Discrete Diffusion Language Models (DLMs) offer a promising non-autoregressive alternative for text generation, yet effective mechanisms for...
arXiv:2606. 15805v1 Announce Type: new Abstract: Discrete diffusion language models enable parallel token generation, offering a pathway to low-latency decoding.
SpecCTMC is a training‑free speculative sampler for masked diffusion language models that reduces the sequential bottleneck of exact continuous‑time sampling. It updates tokens successively using previously computed predictions, evaluates intermediate states in parallel, and applies an exact acceptance‑and‑correction step, thereby preserving the joint distribution of event times, generation order, and token values. Experiments on LLaDA‑8B‑Instruct and Dream‑v0‑Instruct‑7B show 1.73‑4.38× fewer sequential model evaluations while maintaining the reference sampling distribution.
arXiv:2602. 11133v2 Announce Type: replace Abstract: Diffusion language models generate text through iterative refinement, a process that is often computationally inefficient because many tokens reach stability long before the final denoising step.