Distribution Matching Distillation for Continuous Diffusion Language Models
Read the original on arXiv Computation and Language →The Flow has not summarised this story yet — read it at arXiv Computation and Language.
The Flow has not summarised this story yet — read it at arXiv Computation and Language.
The paper introduces DLM-One, a score‑distillation framework that enables one‑step sequence generation with continuous diffusion language models (DLMs). By aligning a student model’s outputs with a pretrained teacher DLM’s score function in the forward‑diffused noisy space, DLM-One removes the need for iterative refinement. Experiments across various DLM architectures show up to ~2000× speedup in sampling steps and ~500× in wall‑clock time while retaining competitive performance, and the authors propose an adversarially‑regularized two‑stage training scheme to mitigate student degeneration.
arXiv:2602. 19066v2 Announce Type: replace-cross Abstract: Diffusion Language Models (DLMs) have recently achieved strong results in text generation.
Consistent Distribution Matching is a data‑free distillation technique that accelerates diffusion and flow models by unifying sample generation and score estimation in a single student network. The method requires only a frozen teacher and a trainable student, optimizing a single objective that provably leads to Wasserstein convergence of the student’s flow‑map to the teacher’s marginals. On ImageNet 256×256, it achieves an FID of 2.04 with one function evaluation and 1.37 with four evaluations after just 40 training epochs, outperforming existing data‑free baselines.
The paper introduces Self-Regularized Flow Language Models (SERF), a self‑supervised framework that enhances continuous diffusion language models by leveraging the denoising trajectory. SERF employs noisy‑to‑clean self‑distillation, where an exponential moving average teacher provides additional supervision along the same interpolation path, and offers two variants—SERF‑L aligning intermediate representations and SERF‑P aligning output token distributions. Experiments on three continuous language‑model paradigms show significant gains in training speed, generation quality, and sampling efficiency, achieving up to 7.7× faster training, 24.3% MAUVE improvement, and 28.8% higher GSM8K accuracy. "whyItMatters":"The results demonstrate that noisy‑to‑clean self‑distillation can substantially improve continuous diffusion language models without external pretraining or inference changes, offering a practical path to faster, higher‑quality language generation."
Flow and diffusion models suffer from slow inference due to computationally expensive numerical integration. Distillation provides a promising way for a student model to learn from a teacher's dynamic...
The paper introduces Trajectory-Regularized Distribution Distillation (TRD), a method that merges distribution-level and trajectory-level supervision for one-step generative models. TRD uses a momentum-mixture policy derived from a single student forward pass, eliminating the need for an extra network to estimate the student’s score. Experiments on SANA-1.6B and Stable Diffusion 3.5 demonstrate that TRD surpasses existing consistency and distribution matching baselines in generation quality and training efficiency, though it slightly reduces sample diversity.