arXiv:2606. 02237v1 Announce Type: new Abstract: Distribution Matching Distillation (DMD) compresses pretrained diffusion models into efficient few-step generators by aligning their noised distributions across all scales.
By Shucheng Li, Iolo Jones, Alexander Tong, Michael M. Bronstein
arXiv:2512.22802v2 Announce Type: replace-cross
Abstract: Step distillation accelerates diffusion sampling by training a few-step student to imitate a many-step teacher, but distillation itself remai...
By Amirhossein Tighkhorshid, Zahra Dehghanian, Hamid R. Rabiee
DMAD (Distribution Matching as Adversarial Distillation) reinterprets distribution matching as a classification task, enabling a few‑step student model to learn log‑density ratios directly via two discriminator heads on a shared backbone. This eliminates the need for an auxiliary diffusion model, reducing memory and computation overhead. Experiments show DMAD achieves state‑of‑the‑art Fréchet Inception Distance scores on ImageNet‑64x64, COCO‑10K, and VBench, and outperforms competing few‑step methods in joint audio‑video generation on MiniMax‑H3‑33B.
By Zhengming Yu, Junkun Yuan, Haotian Yang, Gordon Guocheng Qian, Yizhi Wang, Angtian Wang, Yiding Yang, Bo Liu, Xin Li, Wenping Wang, Chongyang Ma
Consistent Distribution Matching is a data‑free distillation technique that accelerates diffusion and flow models by unifying sample generation and score estimation in a single student network. The method requires only a frozen teacher and a trainable student, optimizing a single objective that provably leads to Wasserstein convergence of the student’s flow‑map to the teacher’s marginals. On ImageNet 256×256, it achieves an FID of 2.04 with one function evaluation and 1.37 with four evaluations after just 40 training epochs, outperforming existing data‑free baselines.
By Yuxiang Fu, Qi Yan, Zike Wu, Yongxing Zhang, Purang Abolmaesumi, Lele Wang, Renjie Liao
arXiv:2607. 02502v1 Announce Type: cross Abstract: On-policy self-distillation (OPSD) has emerged as a practical method for training large language models (LLMs) to reason, where a single model acts as both the teacher and the student with different levels of information access.
By Yunhe Li, Hao Shi, Wenhao Liu, Mengzhe Ruan, Hanxu Hou, Zhongxiang Dai, Shuang Qiu, Linqi Song
The paper introduces pass@k, a metric that evaluates how well generative diffusion models cover their output distribution by measuring the probability that at least one of k independent samples meets a quality criterion. Using this metric, the authors show that while classifier‑free guidance improves single‑draw quality, its advantage diminishes or reverses as k increases, revealing a trade‑off between quality and distribution coverage. They further demonstrate that the choice of training objective in diffusion distillation—whether distribution‑matching or consistency/trajectory‑based—determines whether a few‑step model retains its teacher’s coverage or sacrifices it for higher single‑draw performance, a phenomenon that also appears in few‑step causal video generation.
By Yifei Wang, Xiaoyu Wu, Tsu-Jui Fu, Chen Chen, Liang-Chieh Chen, Zhe Gan, Chen Wei
arXiv:2609.36734v1 Announce Type: new
Abstract: Knowledge Distillation (KD) trains a smaller-capacity student model to imitate a larger-capacity teacher model by matching output distributions, implic...
By Ayan Sengupta, Vaibhav Seth, Tanmoy Chakraborty
arXiv:2610.26025v2 Announce Type: replace-cross
Abstract: Diffusion Transformers (DiTs) deliver high-fidelity image and video generation, but inference requires many sequential model evaluations. Two...
By Shuaiqi Cheng, Jiahao Zhang, Boyuan Gu, Yuxi Chen, Haiying Xu, Zexin Lu, Huiyu Zhou, Xuming Hu
arXiv:2607. 09692v1 Announce Type: new Abstract: Model distillation -- training on outputs from stronger third-party models -- is widely used to boost performance, but raises concerns about unfair advantages and policy violations.
By Rajat Rawat, Sizhe Chen, Akshay Anand, Michael Duan, Bob Rotsted, Sewon Min
On-policy distillation, in which a teacher corrects samples that the student itself generates, presupposes that the two models speak the same language: identical VAE latents, matching architectures, and a common timestep grid. We ask what happens when none of this holds, as when the strongest teacher available and the student one wishes to deploy come from different model families, and find that the standard recipes have no answer: teacher latents cannot serve as targets in a foreign coordinate system, per-pixel losses against a teacher that stochastically re-draws local detail degenerate into blur or divergence, and timestep indices lose their meaning across mismatched schedules.
Flow and diffusion models suffer from slow inference due to computationally expensive numerical integration. Distillation provides a promising way for a student model to learn from a teacher's dynamic...
The paper introduces Velocity Adversarial Distillation (VAD), a method that uses a learned critic to compare velocity fields of student and teacher diffusion models, turning disagreements into learning signals. VAD unifies several existing distillation techniques and leads to VA‑TDM, which applies trajectory distribution matching and transition‑aware online velocity learning to supervise partial denoising along the student’s sampling path. Experiments demonstrate that VA‑TDM achieves state‑of‑the‑art performance for four‑step text‑to‑image and text‑to‑video generation, outperforming prior few‑step methods and approaching the quality of many‑step baselines.
By Yihong Luo, Tianyu Pang, Tianyang Hu