Consistent Distribution Matching is a data‑free distillation technique that accelerates diffusion and flow models by unifying sample generation and score estimation in a single student network. The method requires only a frozen teacher and a trainable student, optimizing a single objective that provably leads to Wasserstein convergence of the student’s flow‑map to the teacher’s marginals. On ImageNet 256×256, it achieves an FID of 2.04 with one function evaluation and 1.37 with four evaluations after just 40 training epochs, outperforming existing data‑free baselines.
By Yuxiang Fu, Qi Yan, Zike Wu, Yongxing Zhang, Purang Abolmaesumi, Lele Wang, Renjie Liao
Flow and diffusion models suffer from slow inference due to computationally expensive numerical integration. Distillation provides a promising way for a student model to learn from a teacher's dynamic...
The paper introduces Phase-wise Velocity Distillation (PVD), a method that splits the diffusion generation timeline into coarse and fine phases, each handled by a half-sized expert. By decoupling structural composition from detail refinement, PVD achieves higher quality image generation within a single full-backbone compute budget, outperforming monolithic student models. Experiments on ImageNet and text-to-image tasks show competitive FID scores and significant reductions in parameters and VRAM usage.
By Zhen Guo, Rongyuan Wu, Qiaosi Yi, Chenxi Xie, Xinyu Wei, Lei Zhang
Omni‑Diffusion‑Distill is a two‑stage distillation framework that compresses unified multimodal diffusion large language models (dLLMs) into a single efficient student while preserving both image generation and multimodal understanding. In the first stage the student learns to skip decoding steps by replaying cached teacher trajectories, and in the second stage it is refined on intermediate states along its own rollouts. The method introduces a pairwise collision penalty to reduce repetition in parallel text decoding and entropy‑matched guidance to prevent entropy collapse in image generation, achieving state‑of‑the‑art trade‑offs: reducing image generation from 128 to 8 steps and multimodal understanding from 512 to 64 steps, yielding 18.2× and 21.2× wall‑clock speedups and strong performance on GenEval, DPG‑Bench, MM‑Vet, and COCO captioning.
By Hong Huang, Chenhongyi Yang, Junzhe Sun, Animesh Sinha, Wuyang Chen, Yifan Jiang
arXiv:2609.37147v1 Announce Type: cross
Abstract: Distributional Diffusion Models (DDMs) replace the standard mean-prediction denoiser with a \emph{distributional} denoiser trained via a scoring rule...
By Tommaso Martorella, Alexandre Galashov, Felix Krause, Stefan Andreas Baumann, Valentin De Bortoli, Arthur Gretton, Bj\"orn Ommer
arXiv:2609.30988v1 Announce Type: new
Abstract: Real-world image super-resolution (SR) requires recovering perceptually realistic high-resolution images from complex low-resolution observations while...
By Xin Di, Mingyu Shi, Yuanfei Bao, Long Peng, Yue Zhao, Jiaming Guo, Renjing Pei, Xueyang Fu, Yang Cao, Zheng-Jun Zha
The paper introduces data‑space iteration, a few‑step generation framework that eliminates the need for a fixed probability‑flow discretization. By refining predictions directly in data space with a shared generator, each iteration is trained to produce the best possible sample within its capacity. Experiments on class‑conditional ImageNet 256×256 show that data‑space iteration outperforms standard discretization baselines and matches or improves upon schedule‑searched variants without schedule‑specific training.
By Shanchuan Lin, Yansong Peng, Fu-Yun Wang, Haoqi Fan
The paper introduces VDOT++, a unified distillation framework for few-step video generation across text-to-video, image-to-video, and condition-based tasks. It extends the previous VDOT method by incorporating an asymmetric unbalanced optimal transport distillation that allows student tokens to carry less mass, and an α1 ground cost that uses a weighted median to preserve modes. The approach combines distribution matching with adversarial refinement and cross-scale distillation, achieving competitive performance on multiple video benchmarks with only four sampling steps.
By Yutong Wang, Xingtong Ge, Enhuai Liu, Yunke Wang, Tianfan Xue, Yu Qiao, Yaohui Wang, Xinyuan Chen, Chang Xu
arXiv:2602.05391v3 Announce Type: replace
Abstract: Dataset distillation seeks to synthesize a compact surrogate dataset that enables performance comparable to training on the original dataset for do...
By Qianxin Xia, Jiawei Du, Yuhan Zhang, Xin Zhang, Xuewan He, Wenbo Jiang, Jielei Wang, Tao Luo, Guoming Lu
arXiv:2609.40235v1 Announce Type: cross
Abstract: Continuous diffusion language models generate all tokens in parallel, yet high-quality generation can still require hundreds of network evaluations (...
By Paul Le Van Kiem, Dario Shariatian, Umut Simsekli, Alain Durmus
The paper introduces IDeaL, a data‑free multi‑teacher distillation technique that generates teacher‑specific, improved samples using decorrelation losses at patch and image levels. By tailoring noise to each teacher, IDeaL produces strong student models that capture complementary teacher information and achieve results close to those distilled from real images. Experiments demonstrate that with only 1,000 images, students trained on IDeaL samples match or exceed the performance of students distilled from a 1,000‑image subset of ImageNet.
By Feyza Yavuz, Mert B\"ulent Sar{\i}y{\i}ld{\i}z, Diane Larlus
arXiv:2606. 25488v1 Announce Type: new Abstract: Knowledge Distillation (KD) is widely used to obtain compact models for efficient inference in resource-constrained environments.
By Yifan Wu, Yiqi Wang, Xichen Ye, Wenjing Yan, Xiaoqiang Li, Cheng Jin, Xiangyu Yue, Weizhong Zhang