arXiv Computer Vision October 16, 2026 By Xinrui Lin, Shumin Wang, Suqi Song, Fan Li, Yanyong Zhang

FocusOPD: Structure-Guided On-Policy Distillation for Diffusion Models

Read the original on arXiv Computer Vision →

FocusOPD introduces a structure‑guided on‑policy distillation method that adds token‑wise directional feature alignment to deep Transformer layers during early denoising steps. This approach addresses the mismatch between teacher and student representations that persists after standard OPD distillation. Experiments on rectified‑flow backbones show that FocusOPD improves performance on GenEval, PickScore, and LAION‑Aesthetic benchmarks, and ablations confirm that selective deep and early supervision is more effective than full‑network alignment.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computer Vision.

arXiv Computer Vision

Sep 25

Beyond Attention Masks: Instruction Anchoring for Efficient In-Context Diffusion Generation

The paper introduces AnchorCache, a parameter‑free token‑layout and attention‑mask design that decouples reference tokens from the target in in‑context diffusion transformers. By inserting static text anchors, the method conditions reference representations on the instruction during cache construction, enabling exact key‑value reuse across denoising steps. To restore quality lost by this structural change, the authors employ teacher‑forced velocity distillation followed by a brief on‑policy stage, achieving full‑attention quality while delivering up to 6.40× speedup in diffusion transformer inference across image, speech, and video benchmarks.

By Yangshuai Liu, Zheming Li, Jiaao Li, Kang He, Ziliang Lai, Zhitai Liu, Chengru Song

llms diffusion efficiency benchmarks

arXiv Computer Vision

Oct 28

Nemotron-Labs-Diffusion-Uni: Elastic Self-Refining Unified Diffusion LLM with Cross-Paradigm On-Policy Distillation

Nemotron‑Labs‑Diffusion‑Uni (NLD‑Uni) is a unified discrete diffusion large language model designed for visual understanding and generation. It introduces a cross‑paradigm on‑policy distillation framework that transfers capabilities from autoregressive VLMs and continuous flow models into a single DLLM, and a progressive self‑refining diffusion process that corrupts tokens gradually during training. Experiments show NLD‑Uni outperforms other unified DLLMs on tasks such as visual question answering, multi‑modal reasoning, object grounding, text‑to‑image generation, and image editing.

By Shufan Li, Hanrong Ye, Greg Heinrich, Yonggan Fu, Haotian Zhang, Sifei Liu, Aditya Grover, Pavlo Molchanov

llms diffusion efficiency multimodal benchmarks

Hugging Face Trending Papers

Aug 11

Grid-Preserving Knowledge Distillation: Transferring Convolutional Inductive Bias to Vision Transformers under Data Scarcity

Vision Transformers underperform convolutional networks when training data is scarce, and distilling convolutional inductive biases from a CNN teacher is an effective remedy that leaves the deployed model unchanged. General-purpose feature distillation, however, transfers little in this setting.

llms efficiency benchmarks safety

arXiv Machine Learning

Sep 15

Data Efficient Any Transformer-to-Mamba Distillation via Attention Bridge

arXiv:2510.19266v3 Announce Type: replace Abstract: State-space models (SSMs) have emerged as promising alternatives to Transformers for sequence modeling. However, training competitive SSMs from scr...

By Penghao Wang, Yuhao Zhou, Mengxuan Wu, Panpan Zhang, Zhangyang Wang, Kai Wang

llms computer-vision efficiency safety

arXiv AI

Sep 15

Qwen-Image-Flash: Rethinking the Training Recipe for Few-Step Distillation

arXiv:2606.03746v3 Announce Type: replace-cross Abstract: Few-step distillation has emerged as a critical component in the development of advanced visual generative foundation models, substantially r...

By Tianhe Wu, Zikai Zhou, Kun Yan, Kaiyuan Gao, Lihan Jiang, Jiahao Li, Jie Zhang, Ningyuan Tang, Shengming Yin, Xiaoyue Chen, Xiao Xu, Yilei Chen, Yuxiang Chen, Yan Shu, Yixian Xu, Yanran Zhang, Zihao Liu, Zhendong Wang, Zekai Zhang, Deqing Li, Liang Peng, Yi Wang, Zeke Xie, Jingren Zhou, Bo Zheng, Chenfei Wu

diffusion efficiency

arXiv AI

Jun 2

DASH: Dual-Branch Score Distillation for Guidance-Calibrated Compact Diffusion Models

arXiv:2606. 00798v1 Announce Type: cross Abstract: Parameter compression of class-conditional diffusion models reveals an underexplored limitation in output-level distillation: the unconditional score branch remains unsupervised, leaving the classifier-free guidance gap underdetermined in the student.

By Abdullah Al Shafi, Kazi Saeed Alam, Sk Imran Hossain, Engelbert Mephu Nguifo

diffusion efficiency benchmarks