arXiv Machine Learning
CARD: Closing the Teacher-Student Loop for Adversarial Robustness Distillation
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Computer Vision
Oct 9
Collaboratively Guided Adversarial Robust Distillation with Teacher-Favorable Examples
Collaboratively Guided Adversarial Robust Distillation (CGARD) is a method that transfers robustness from high‑capacity teachers to compact students by jointly optimizing student‑adversarial and teacher‑collaborative examples within the same perturbation neighborhood. The teacher‑collaborative example is constrained so that the teacher’s cross‑entropy loss does not exceed that of the clean input, combining collaborative guidance with adversarial supervision. Experiments on CIFAR‑10 and CIFAR‑100, including white‑box and black‑box evaluations, show consistent robustness improvements over strong adversarial distillation baselines.
arXiv Machine Learning
Jul 31
Improving the Robustness/Accuracy Tradeoff Against Adversarial Attacks Using Information Bottleneck Distillation Through Dual Teachers
arXiv:2607. 27737v1 Announce Type: new Abstract: Deep neural networks (DNNs) have achieved remarkable success in classical machine learning problems.
arXiv Machine Learning
Jul 20
SODA: Semi On-Policy Black-Box Distillation for Large Language Models
arXiv:2604. 03873v4 Announce Type: replace Abstract: Black-box knowledge distillation for large language models presents a strict trade-off.
arXiv Machine Learning
Jun 11
Beyond Dark Knowledge: Mixup-Based Distillation for Reliable Predictions
arXiv:2606. 12171v1 Announce Type: cross Abstract: Knowledge Distillation (KD) and mixup have proven effective at inducing smoothness in class boundaries; KD captures inherent class relationships in probability distributions, and mixup enforces them through convex combinations of inputs.
arXiv AI
Sep 10
How to Backdoor Image Knowledge Distillation
The paper demonstrates that image knowledge distillation can be backdoored even when the teacher model is clean, by poisoning the distillation dataset with triggered and manipulated images that the teacher already classifies as a target label. The attack, effective at poisoning rates as low as 10%, uses targeted adversarial perturbations and GAN-based class transitions to embed a backdoor into the student model while preserving its performance on clean data. The study highlights that the security of knowledge distillation depends not only on the teacher but also on the integrity of the distillation data.
arXiv Machine Learning
Jul 30
Weak-to-Strong On-Policy Distillation
arXiv:2607. 26246v1 Announce Type: new Abstract: On-policy distillation (OPD), which aligns a student with the teacher's token-level distribution on the student's own rollouts, is an effective paradigm for transferring capabilities across LLMs.