The paper introduces DLM-One, a score‑distillation framework that enables one‑step sequence generation with continuous diffusion language models (DLMs). By aligning a student model’s outputs with a pretrained teacher DLM’s score function in the forward‑diffused noisy space, DLM-One removes the need for iterative refinement. Experiments across various DLM architectures show up to ~2000× speedup in sampling steps and ~500× in wall‑clock time while retaining competitive performance, and the authors propose an adversarially‑regularized two‑stage training scheme to mitigate student degeneration.
By Tianqi Chen, Shujian Zhang, Mingyuan Zhou
arXiv:2602. 19066v2 Announce Type: replace-cross Abstract: Diffusion Language Models (DLMs) have recently achieved strong results in text generation.
By David Li, Nikita Gushchin, Dmitry Abulkhanov, Eric Moulines, Ivan Oseledets, Maxim Panov, Alexander Korotin
Consistent Distribution Matching is a data‑free distillation technique that accelerates diffusion and flow models by unifying sample generation and score estimation in a single student network. The method requires only a frozen teacher and a trainable student, optimizing a single objective that provably leads to Wasserstein convergence of the student’s flow‑map to the teacher’s marginals. On ImageNet 256×256, it achieves an FID of 2.04 with one function evaluation and 1.37 with four evaluations after just 40 training epochs, outperforming existing data‑free baselines.
By Yuxiang Fu, Qi Yan, Zike Wu, Yongxing Zhang, Purang Abolmaesumi, Lele Wang, Renjie Liao
The paper introduces Self-Regularized Flow Language Models (SERF), a self‑supervised framework that enhances continuous diffusion language models by leveraging the denoising trajectory. SERF employs noisy‑to‑clean self‑distillation, where an exponential moving average teacher provides additional supervision along the same interpolation path, and offers two variants—SERF‑L aligning intermediate representations and SERF‑P aligning output token distributions. Experiments on three continuous language‑model paradigms show significant gains in training speed, generation quality, and sampling efficiency, achieving up to 7.7× faster training, 24.3% MAUVE improvement, and 28.8% higher GSM8K accuracy.
"whyItMatters":"The results demonstrate that noisy‑to‑clean self‑distillation can substantially improve continuous diffusion language models without external pretraining or inference changes, offering a practical path to faster, higher‑quality language generation."
By Theodoros Kouzelis, Spyros Gidaris, Nikos Komodakis
Flow and diffusion models suffer from slow inference due to computationally expensive numerical integration. Distillation provides a promising way for a student model to learn from a teacher's dynamic...
The paper introduces Trajectory-Regularized Distribution Distillation (TRD), a method that merges distribution-level and trajectory-level supervision for one-step generative models. TRD uses a momentum-mixture policy derived from a single student forward pass, eliminating the need for an extra network to estimate the student’s score. Experiments on SANA-1.6B and Stable Diffusion 3.5 demonstrate that TRD surpasses existing consistency and distribution matching baselines in generation quality and training efficiency, though it slightly reduces sample diversity.
By An Zhao, Yixiang Zhou, Zejian Li, Yi Guo, Pei Chen, Lingyun Sun
The paper introduces a new method for training data attribution in diffusion models called TID, which uses a local score discrepancy measure and can be estimated without retraining. It further distills this approach into TIDE, a forward‑only student that reproduces the teacher’s rankings using internal activations, achieving comparable accuracy at dramatically lower query cost. Experiments on CIFAR‑10, ArtBench‑10, and MS‑COCO show that TID outperforms existing methods and TIDE attributes samples in milliseconds, faster than generation itself.
By Shixuan Liu, Joan Serr\`a, Kin Wai Cheuk, Jinju Kim, Woosung Choi, Yukara Ikemiya, Wei-Hsiang Liao, Jiaqi W. Ma, Yuki Mitsufuji
arXiv:2606. 29869v1 Announce Type: cross Abstract: Knowledge distillation (KD) is a key technique for compressing Large Language Models (LLMs), yet methods relying on a single KL objective often fail to balance primary distribution fitting with long-tail probability modeling, limiting both generation quality and generalization.
By Zilong Liu, Xuewen Zhang, Jinrui Xing, Juyi Qiao, Huiyong Wang, Junming Jiao
arXiv:2610.19305v1 Announce Type: new
Abstract: One-step diffusion distillation achieves strong performance on small datasets but remains insufficient for high-resolution or complex text-to-image set...
By Ngoc Duy Tran, Kien Do, Duc Thanh Nguyen, Truyen Tran
arXiv:2610.07706v1 Announce Type: cross
Abstract: Autoregressive large language models (LLMs) have rapidly advanced in capability, but their increasing scale comes with substantial computational and...
By Byeonghu Na, Donghyeok Shin, Yeongmin Kim, Mina Kang, Il-Chul Moon
LangFlow is a continuous diffusion language model that bridges embedding-space diffusion with Flow Matching, introducing three key innovations: an ODE-based NLL bound, an information-uniform noise schedule using a learnable Gumbel distribution, and self-conditioning training. These advances enable LangFlow to match or surpass top discrete diffusion models in generative quality, achieving perplexities of 92.3 on LM1B and 36.5 on OpenWebText, and outperforming discrete counterparts on reasoning tasks such as Sudoku. The work demonstrates that continuous diffusion can rival discrete diffusion for language modeling, offering a promising new paradigm.
By Yuxin Chen, Chumeng Liang, Hangke Sui, Ruihan Guo, Chaoran Cheng, Jiaxuan You, Ge Liu
arXiv:2603. 22216v2 Announce Type: replace-cross Abstract: The slow, sequential nature of autoregressive (AR) language models has driven the adoption of parallel decoding methods.
By Chi Zhang, Xixi Hu, Bo Liu, Qiang Liu