AURORA-LM is a continuous‑latent diffusion language model that separates the creation of a high‑capacity, decodable text representation from the modeling of its distribution. It uses a Query‑based Encoder‑Decoder to produce a prefix‑aligned latent sequence and a Block‑Causal Diffusion Transformer to learn its distribution via flow matching, generating text block‑by‑block while denoising positions in parallel. The model calibrates noise levels to latent width, introduces self‑trajectory consistency, and achieves state‑of‑the‑art performance on OpenWebText generation and XSum summarization, outperforming larger publicly released latent‑diffusion models when scaled to 1 B parameters.
arXiv:2508. 10875v3 Announce Type: replace-cross Abstract: Diffusion Language Models (DLMs) are rapidly emerging as a powerful and promising alternative to the dominant autoregressive (AR) paradigm.
The paper introduces Hierarchical Continuous Diffusion Language Models (H-CDLMs), a framework that jointly diffuses multiple token modalities—individual tokens and coarser clusters of token embeddings—to enhance continuous diffusion language models. Applied to the CoBit architecture, the resulting H-CoBit achieves significant empirical gains, improving MAUVE scores and achieving lower generative perplexity on LM1B and OWT, while also outperforming prior continuous diffusion models on GSM8K. The approach generalizes to other continuous generative paradigms, as shown by consistent improvements when applied to the flow matching model FLM.
The paper revisits pixel-space diffusion models, showing that a vanilla diffusion transformer can generate directly in pixel space by swapping velocity prediction for clean image prediction. It introduces MFlow, an adaptive parameterization that interpolates between these two prediction modes based on data representation and model capacity, and demonstrates competitive performance on ImageNet, UCF‑101, and text‑to‑image tasks. Additionally, the authors add ConvRefiner, a lightweight convolutional module trained post‑hoc to restore high‑frequency details while preserving FID.
arXiv:2609.37080v1 Announce Type: new Abstract: Latent Diffusion Models (LDMs) typically adopt a two-stage pipeline: an auto-encoder (AE) is first pre-trained to define a latent space, then a diffusi...
arXiv:2605. 00161v2 Announce Type: replace Abstract: Diffusion language models (DLMs) are an attractive alternative to autoregressive models because they promise sublinear-time, parallel generation, yet practical gains remain elusive as high-quality samples still demand hundreds of refinement steps.
The paper introduces Representation-based Masked Diffusion Model (RMDM), a new framework for language modeling that improves upon existing Masked Diffusion Models by incorporating global semantic guidance. RMDM encodes text into a continuous semantic space with a pretrained encoder, normalizes this representation to a Gaussian prior via an invertible transformation, and then trains a masked diffusion model conditioned on this latent representation to coordinate parallel token updates. Experiments show that RMDM yields higher generation quality, especially when using aggressive few‑step sampling.
The paper introduces Self-Regularized Flow Language Models (SERF), a self‑supervised framework that enhances continuous diffusion language models by leveraging the denoising trajectory. SERF employs noisy‑to‑clean self‑distillation, where an exponential moving average teacher provides additional supervision along the same interpolation path, and offers two variants—SERF‑L aligning intermediate representations and SERF‑P aligning output token distributions. Experiments on three continuous language‑model paradigms show significant gains in training speed, generation quality, and sampling efficiency, achieving up to 7.7× faster training, 24.3% MAUVE improvement, and 28.8% higher GSM8K accuracy. "whyItMatters":"The results demonstrate that noisy‑to‑clean self‑distillation can substantially improve continuous diffusion language models without external pretraining or inference changes, offering a practical path to faster, higher‑quality language generation."
The paper introduces Sigma, a large-scale continuous diffusion language model (3B/8B parameters) that uses steerable, low-dimensional ODE/SDE latent trajectories to address the non-smoothness of discrete diffusion models. Sigma is trained blockwise via likelihood optimization, jointly denoises Gaussian-corrupted token embeddings, and learns an optimal embedding geometry, leveraging pre-trained autoregressive weights for faster training. During inference, classifier-free guidance and score temperature are identified as essential for high-fidelity reasoning and coding, and Sigma matches or exceeds discrete models on benchmarks such as GSM8K, Minerva, HumanEval, MBPP, MATH-500, and AIME, while also revealing unique structural benefits like embedding-space steering and graceful degradation for low NFEs.
arXiv:2610.08108v1 Announce Type: new Abstract: Diffusion language models (dLLMs) have emerged as a promising alternative to autoregressive (AR) language models, offering flexible token-update orders...
arXiv:2602.10216v2 Announce Type: replace Abstract: A single text prompt passed to a diffusion model yields a wide range of visual outputs determined solely by a stochastic process, leaving users wit...
Zarya is a hybrid language model that jointly trains an autoregressive objective and a masked-diffusion objective within a single architecture. It structures training data into variable-size slots and uses a curriculum that gradually increases slot granularity, allowing a smooth transition from fine-grained AR learning to coarse-grained diffusion learning. At inference, Zarya offers two decoding modes—MDM sampling with first-hitting denoising and slotted speculative decoding that interleaves diffusion-based selection with autoregressive infilling—while fully decoupling training and inference regimes and supporting extensive configurability.
arXiv:2306.08757v2 Announce Type: replace-cross Abstract: While diffusion models excel at generating high-quality samples, their latent variables typically lack semantic meaning and are not suitable...