Hugging Face Trending Papers

Disagreement-Aware Distribution Matching Distillation for Few-Step Diffusion Models

Disagreement‑Aware Distribution Matching Distillation (DA‑DMD) builds on Distribution Matching Distillation (DMD) for few‑step diffusion models by leveraging the teacher‑fake disagreement signal already computed during training. The method activates only in the high‑noise regime, modulating update strength based on disagreement magnitude, and introduces no extra models or inference changes. Experiments across diffusion backbones show consistent improvements in few‑step generation, with DA‑DMD boosting HPSv2 on SD3.5‑L from 28.23 to 30.93 and achieving a 64.4% average win rate over DMD2.

arXiv AI
Oct 2

DMAD: Distribution Matching as Adversarial Distillation for Fast Visual Generation

DMAD (Distribution Matching as Adversarial Distillation) reinterprets distribution matching as a classification task, enabling a few‑step student model to learn log‑density ratios directly via two discriminator heads on a shared backbone. This eliminates the need for an auxiliary diffusion model, reducing memory and computation overhead. Experiments show DMAD achieves state‑of‑the‑art Fréchet Inception Distance scores on ImageNet‑64x64, COCO‑10K, and VBench, and outperforms competing few‑step methods in joint audio‑video generation on MiniMax‑H3‑33B.

By Zhengming Yu, Junkun Yuan, Haotian Yang, Gordon Guocheng Qian, Yizhi Wang, Angtian Wang, Yiding Yang, Bo Liu, Xin Li, Wenping Wang, Chongyang Ma
arXiv Machine Learning
Oct 8

Consistent Distribution Matching for Data-Free Diffusion Distillation

Consistent Distribution Matching is a data‑free distillation technique that accelerates diffusion and flow models by unifying sample generation and score estimation in a single student network. The method requires only a frozen teacher and a trainable student, optimizing a single objective that provably leads to Wasserstein convergence of the student’s flow‑map to the teacher’s marginals. On ImageNet 256×256, it achieves an FID of 2.04 with one function evaluation and 1.37 with four evaluations after just 40 training epochs, outperforming existing data‑free baselines.

By Yuxiang Fu, Qi Yan, Zike Wu, Yongxing Zhang, Purang Abolmaesumi, Lele Wang, Renjie Liao
arXiv AI
Jul 3

DemoPSD: Disagreement-Modulated Policy Self-Distillation

arXiv:2607. 02502v1 Announce Type: cross Abstract: On-policy self-distillation (OPSD) has emerged as a practical method for training large language models (LLMs) to reason, where a single model acts as both the teacher and the student with different levels of information access.

By Yunhe Li, Hao Shi, Wenhao Liu, Mengzhe Ruan, Hanxu Hou, Zhongxiang Dai, Shuang Qiu, Linqi Song
arXiv Machine Learning
Oct 2

Visualizing Distribution Coverage in Generative Diffusion Models

The paper introduces pass@k, a metric that evaluates how well generative diffusion models cover their output distribution by measuring the probability that at least one of k independent samples meets a quality criterion. Using this metric, the authors show that while classifier‑free guidance improves single‑draw quality, its advantage diminishes or reverses as k increases, revealing a trade‑off between quality and distribution coverage. They further demonstrate that the choice of training objective in diffusion distillation—whether distribution‑matching or consistency/trajectory‑based—determines whether a few‑step model retains its teacher’s coverage or sacrifices it for higher single‑draw performance, a phenomenon that also appears in few‑step causal video generation.

By Yifei Wang, Xiaoyu Wu, Tsu-Jui Fu, Chen Chen, Liang-Chieh Chen, Zhe Gan, Chen Wei
arXiv Machine Learning
Jul 14

Reference-Based Distillation Detection in LLMs

arXiv:2607. 09692v1 Announce Type: new Abstract: Model distillation -- training on outputs from stronger third-party models -- is widely used to boost performance, but raises concerns about unfair advantages and policy violations.

By Rajat Rawat, Sizhe Chen, Akshay Anand, Michael Duan, Bob Rotsted, Sewon Min
Hugging Face Trending Papers
Aug 4

Any-OPD: Heterogeneous On-Policy Distillation for Flow-Matching Models via Representation-Space Bridging

On-policy distillation, in which a teacher corrects samples that the student itself generates, presupposes that the two models speak the same language: identical VAE latents, matching architectures, and a common timestep grid. We ask what happens when none of this holds, as when the strongest teacher available and the student one wishes to deploy come from different model families, and find that the standard recipes have no answer: teacher latents cannot serve as targets in a foreign coordinate system, per-pixel losses against a teacher that stochastically re-draws local detail degenerate into blur or divergence, and timestep indices lose their meaning across mismatched schedules.

arXiv Computer Vision
Nov 5

VA-TDM: A Unified View for Improving Few-Step Diffusion Models by Velocity Adversarial Distillation

The paper introduces Velocity Adversarial Distillation (VAD), a method that uses a learned critic to compare velocity fields of student and teacher diffusion models, turning disagreements into learning signals. VAD unifies several existing distillation techniques and leads to VA‑TDM, which applies trajectory distribution matching and transition‑aware online velocity learning to supervise partial denoising along the student’s sampling path. Experiments demonstrate that VA‑TDM achieves state‑of‑the‑art performance for four‑step text‑to‑image and text‑to‑video generation, outperforming prior few‑step methods and approaching the quality of many‑step baselines.

By Yihong Luo, Tianyu Pang, Tianyang Hu