The paper introduces Spherical Brownian Diffusion, a new continuous diffusion language model that uses Brownian motion on the unit sphere as its forward process. It provides a principled heat semigroup, an additive dimension‑free noise clock, and an exact ancestral bridge, while a clean‑token posterior learned with cross‑entropy enables multiple samplers. Experiments on Sudoku and GSM8K show that the choice of corruption process has little effect, whereas sampler selection can significantly impact performance, with memory and token‑selection effects differing across tasks.
arXiv:2606. 15327v1 Announce Type: new Abstract: Diffusion Language Models (DLMs) have demonstrated strong scaling capacity as alternatives to autoregressive language models.
arXiv:2607. 24507v1 Announce Type: cross Abstract: Existing methods mainly adapt pretrained autoregressive (AR) language models to masked diffusion, whereas we directly adapt them to uniform-noise diffusion, where every token remains editable during sampling.
The paper investigates two key design levers for Continuous Diffusion Language Models (CDLMs): self‑conditioning and embedding geometry. It shows that self‑conditioning mitigates a train/inference mismatch and can be implemented efficiently via a two‑level interpolant called DoubleCat, which significantly boosts TinyGSM accuracy. By adding discrete reveals to DoubleCat, the authors create SemiCat, achieving discrete‑diffusion‑level performance and outperforming discrete diffusion on several likelihood‑based benchmarks at 1.7B parameters. The study also finds that cross‑entropy training drives embeddings toward a geometry with decoding error rates comparable to one‑hot representations, and briefly explores additional options such as hybrid corruption processes and asynchronous noise schedules.
arXiv:2605. 00161v2 Announce Type: replace Abstract: Diffusion language models (DLMs) are an attractive alternative to autoregressive models because they promise sublinear-time, parallel generation, yet practical gains remain elusive as high-quality samples still demand hundreds of refinement steps.
Forward-Free Diffusion Language Models with BPTT-Free Looped Refinement (FReDA) removes the need for a hand‑designed forward process in diffusion language modeling by treating model‑generated drafts as implicit intermediate states and refining them recursively. The approach detaches earlier refinement passes, backpropagating only through the final pass, and supports both self‑refinement and Best‑of‑N candidate selection. In sub‑8B experiments, FReDA‑4B surpasses larger diffusion baselines on reasoning and coding tasks, achieving up to 15% absolute gains and a 1.5‑1.8× speedup while scaling well with additional refinement steps.
arXiv:2609.10305v1 Announce Type: new Abstract: Language models under one million parameters matter for edge deployment, domain adaptation, and reproducible research, yet a two-layer LSTM or Transfor...
Diffusion language models (dLLMs) have emerged as a promising alternative to autoregressive (AR) language models, offering flexible token-update orders and parallel decoding. Recent dLLMs are often in...
Hierarchical Continuous Diffusion Language Models (HC‑DLM) combine discrete token generation with a continuous latent trajectory in a single denoising process, addressing the independence bottleneck of parallel decoding in discrete diffusion models. The training objective is derived from a variational bound on token likelihood, and the latent state is the sole persistent generative element, with tokens read out and fed back at each step. Experiments on Sudoku, Countdown, and LM1B show HC‑DLM outperforming both discrete and continuous diffusion baselines in puzzle accuracy and generative perplexity.
The paper introduces Corrective Diffusion Language Models (CDLMs), which enhance masked diffusion language models by explicitly supervising visible corrupted tokens to enable confidence-guided refinement of incorrect tokens. Experiments show that this correction-oriented post‑training improves iterative refinement and parallel decoding across models ranging from 0.5B to 8B, outperforming standard masked‑diffusion fine‑tuning on benchmarks such as Code Revision, HumanEval(+), and MATH. The authors also provide a new Code Revision Benchmark and release code and data for reproducibility.
The paper introduces a lightweight single‑layer sampler that allows masked diffusion language models to approximate joint sampling of multiple tokens in a single full‑model forward pass. By training the sampler to mimic exact joint sampling from a frozen diffusion model, the authors enable parallel unmasking of tokens while maintaining a close match to the true joint distribution. Experiments on Dream‑7B and Llada‑7B models show that unmasking four tokens per denoising step yields a MAUVE score of 0.87, a substantial improvement over the marginal baseline of 0.31.
arXiv:2607. 14106v1 Announce Type: cross Abstract: In this paper we introduce token time continuous diffusion (TTCD), a new diffusion language model which (a) operates in continuous space, deterministically mapping Gaussian noise to a final token canvas with no further sampling, and crucially (b) incorporates a new notion of per-token times, with some tokens proceeding from noise to token at a faster rate than others.
arXiv:2606. 11552v1 Announce Type: cross Abstract: Large language models (LLMs) achieve remarkable performance across a wide range of tasks, but their autoregressive decoding process incurs substantial inference costs due to inherently sequential token generation.