The paper introduces AnchorCache, a parameter‑free token‑layout and attention‑mask design that decouples reference tokens from the target in in‑context diffusion transformers. By inserting static text anchors, the method conditions reference representations on the instruction during cache construction, enabling exact key‑value reuse across denoising steps. To restore quality lost by this structural change, the authors employ teacher‑forced velocity distillation followed by a brief on‑policy stage, achieving full‑attention quality while delivering up to 6.40× speedup in diffusion transformer inference across image, speech, and video benchmarks.
By Yangshuai Liu, Zheming Li, Jiaao Li, Kang He, Ziliang Lai, Zhitai Liu, Chengru Song
Nemotron‑Labs‑Diffusion‑Uni (NLD‑Uni) is a unified discrete diffusion large language model designed for visual understanding and generation. It introduces a cross‑paradigm on‑policy distillation framework that transfers capabilities from autoregressive VLMs and continuous flow models into a single DLLM, and a progressive self‑refining diffusion process that corrupts tokens gradually during training. Experiments show NLD‑Uni outperforms other unified DLLMs on tasks such as visual question answering, multi‑modal reasoning, object grounding, text‑to‑image generation, and image editing.
By Shufan Li, Hanrong Ye, Greg Heinrich, Yonggan Fu, Haotian Zhang, Sifei Liu, Aditya Grover, Pavlo Molchanov
Vision Transformers underperform convolutional networks when training data is scarce, and distilling convolutional inductive biases from a CNN teacher is an effective remedy that leaves the deployed model unchanged. General-purpose feature distillation, however, transfers little in this setting.
arXiv:2510.19266v3 Announce Type: replace
Abstract: State-space models (SSMs) have emerged as promising alternatives to Transformers for sequence modeling. However, training competitive SSMs from scr...
By Penghao Wang, Yuhao Zhou, Mengxuan Wu, Panpan Zhang, Zhangyang Wang, Kai Wang
arXiv:2606.03746v3 Announce Type: replace-cross
Abstract: Few-step distillation has emerged as a critical component in the development of advanced visual generative foundation models, substantially r...
By Tianhe Wu, Zikai Zhou, Kun Yan, Kaiyuan Gao, Lihan Jiang, Jiahao Li, Jie Zhang, Ningyuan Tang, Shengming Yin, Xiaoyue Chen, Xiao Xu, Yilei Chen, Yuxiang Chen, Yan Shu, Yixian Xu, Yanran Zhang, Zihao Liu, Zhendong Wang, Zekai Zhang, Deqing Li, Liang Peng, Yi Wang, Zeke Xie, Jingren Zhou, Bo Zheng, Chenfei Wu
arXiv:2606. 00798v1 Announce Type: cross Abstract: Parameter compression of class-conditional diffusion models reveals an underexplored limitation in output-level distillation: the unconditional score branch remains unsupervised, leaving the classifier-free guidance gap underdetermined in the student.
By Abdullah Al Shafi, Kazi Saeed Alam, Sk Imran Hossain, Engelbert Mephu Nguifo
arXiv:2605.06597v3 Announce Type: replace-cross
Abstract: Self-distillation (SD) offers a promising path for adapting large language models (LLMs) without relying on stronger external teachers. Howev...
By Yiqiao Jin, Yiyang Wang, Lucheng Fu, Yijia Xiao, Yinyi Luo, Haoxin Liu, B. Aditya Prakash, Chao Zhang, Josiah Hester, Jindong Wang, Srijan Kumar
arXiv:2611.06796v1 Announce Type: new
Abstract: Few-step distillation offers efficient generation with diffusion and flow models, but preserving quality with only a few sampling steps remains challen...
By Yihong Luo, Tianyu Pang, Tianyang Hu
arXiv:2606. 23898v1 Announce Type: cross Abstract: Distilling conditional diffusion models aims to transfer the behavior of a large teacher to a smaller student while preserving alignment across conditioning inputs.
By Loay Mualem, Vinh Tong, Samir Darouich, Mathias Niepert
OptIMerge is an optimization-based framework that merges multiple image‑to‑image diffusion models fine‑tuned for dense, pixel‑accurate tasks. Instead of naive scalar averaging, it learns lightweight, architecture‑aware merger coefficients—such as low‑rank feature alignment for attention layers and per‑kernel coefficients for convolutional layers—by minimizing a multi‑task diffusion loss on a small image set. The approach keeps the expert models frozen, uses less than 0.4% of the network parameters, and achieves multi‑task performance comparable to joint training while outperforming zero‑shot model soups.
By Viraj Shah, Ashwini Pokle, Peyman Milanfar, Mauricio Delbracio, Hossein Talebi
The paper investigates how knowledge distillation (KD) applied at intermediate layers of a neural network can affect overfitting and model performance. While traditional KD focuses on the final output, this study explores block‑wise KD across eleven datasets, finding that on standard datasets the last block suffices, but on fine‑grained, data‑scarce settings intermediate supervision significantly improves accuracy. The authors also analyze optimal supervision granularity using attention maps, Centered Kernel Alignment, and Grad‑CAM, and examine teacher‑student fine‑tuning strategies.
By Irene Trigueros-Lorca, Leonardo Concepci\'on, Christian Wagner, Isaac Triguero, Daniel Molina
arXiv:2609.40305v1 Announce Type: new
Abstract: Improving text-to-image models has traditionally relied on increasing model size or the number of denoising steps. In this work, we explore an alternat...
By Yong Xien Chng, Tianyi Chen, Wenwen Tong, Haiwen Diao, Zhongang Cai, Lei Yang, Ziwei Liu, Lewei Lu, Dahua Lin, Gao Huang