Collaboratively Guided Adversarial Robust Distillation (CGARD) is a method that transfers robustness from high‑capacity teachers to compact students by jointly optimizing student‑adversarial and teacher‑collaborative examples within the same perturbation neighborhood. The teacher‑collaborative example is constrained so that the teacher’s cross‑entropy loss does not exceed that of the clean input, combining collaborative guidance with adversarial supervision. Experiments on CIFAR‑10 and CIFAR‑100, including white‑box and black‑box evaluations, show consistent robustness improvements over strong adversarial distillation baselines.
By Zhi Li, Haowei Liu, Hongchen Yang, Xiaoxuan Wang, Song Gao, Shaowen Yao, Wei Zhou
arXiv:2607. 27737v1 Announce Type: new Abstract: Deep neural networks (DNNs) have achieved remarkable success in classical machine learning problems.
By Vincent Ryusuke Takahashi, Yoshinari Takeishi, Jun'ichi Takeuchi, Kave Salamatian
arXiv:2604. 03873v4 Announce Type: replace Abstract: Black-box knowledge distillation for large language models presents a strict trade-off.
By Xiwen Chen, Jingjing Wang, Wenhui Zhu, Peijie Qiu, Xuanzhao Dong, Yueyue Deng, Hejian Sang, Zhipeng Wang, Alborz Geramifard, Feng Luo
arXiv:2606. 12171v1 Announce Type: cross Abstract: Knowledge Distillation (KD) and mixup have proven effective at inducing smoothness in class boundaries; KD captures inherent class relationships in probability distributions, and mixup enforces them through convex combinations of inputs.
By Jos\'e Medina, Paul Honeine, Abdelaziz Bensrhair, Amnir Hadachi
The paper demonstrates that image knowledge distillation can be backdoored even when the teacher model is clean, by poisoning the distillation dataset with triggered and manipulated images that the teacher already classifies as a target label. The attack, effective at poisoning rates as low as 10%, uses targeted adversarial perturbations and GAN-based class transitions to embed a backdoor into the student model while preserving its performance on clean data. The study highlights that the security of knowledge distillation depends not only on the teacher but also on the integrity of the distillation data.
By Qian Ma, Chen Wu, Prasenjit Mitra, Sencun Zhu
arXiv:2607. 26246v1 Announce Type: new Abstract: On-policy distillation (OPD), which aligns a student with the teacher's token-level distribution on the student's own rollouts, is an effective paradigm for transferring capabilities across LLMs.
By Fangxu Yu, Zinan Lin, Xiaodong Liu, Weijia Xu, Michael Xu, Tianyi Zhou, Jianfeng Gao
arXiv:2609.36734v1 Announce Type: new
Abstract: Knowledge Distillation (KD) trains a smaller-capacity student model to imitate a larger-capacity teacher model by matching output distributions, implic...
By Ayan Sengupta, Vaibhav Seth, Tanmoy Chakraborty
arXiv:2606. 31653v1 Announce Type: cross Abstract: Certified training aims to produce models whose predictions can be formally verified against adversarial perturbations, typically by optimising upper bounds on the worst-case loss over an allowed perturbation set.
By Matteo Melis, Jesus Martinez Del Rincon, Vishal Sharma
arXiv:2609.13199v1 Announce Type: new
Abstract: Knowledge distillation aims to improve the performance of lightweight student models by transferring knowledge from larger and more powerful teacher mo...
By Dawen Jiang, Zhishu Shen, Zeyu Liu, Tiehua Zhang
arXiv:2609.08798v1 Announce Type: new
Abstract: Weak-to-strong generalization asks whether stronger models can learn from weaker supervisors and surpass them. This question is particularly important...
By Youngrok Park, Sangmin Bae, Hojung Jung, Jongwoo Ko, Yunseon Choi, Young Jin Kim, Pashmina Cameron, Aaron Courville, Se-Young Yun
arXiv:2609.22566v1 Announce Type: cross
Abstract: Knowledge distillation (KD) aims to compress high-performance teacher LLMs into lightweight students. However, distilled students often exhibit subst...
By Dileesha Kannangara, Sanghamitra Dutta
The paper introduces a recursive self-improvement framework for language models that replaces an external teacher with a frozen copy of the student, enabling dynamic co-evolution (DCE) and self-refined concise learning (SRCL). DCE allows the privileged teacher to evolve alongside the student, while SRCL trains on shorter, verified rewrites to reduce verbosity. Experiments show that the combined DCE+SRCL approach outperforms traditional on‑policy self‑distillation across multiple model sizes and math benchmarks, achieving significant accuracy gains and shorter outputs.
By Shangjian Yin, Zehao Zhao, Kavosh Asadi, Rui Liu, Yuchen Lu, Shike Mei, Hang Cui, Luke Simon, Zhouxing Shi, Hamed Firooz