Collaboratively Guided Adversarial Robust Distillation (CGARD) is a method that transfers robustness from high‑capacity teachers to compact students by jointly optimizing student‑adversarial and teacher‑collaborative examples within the same perturbation neighborhood. The teacher‑collaborative example is constrained so that the teacher’s cross‑entropy loss does not exceed that of the clean input, combining collaborative guidance with adversarial supervision. Experiments on CIFAR‑10 and CIFAR‑100, including white‑box and black‑box evaluations, show consistent robustness improvements over strong adversarial distillation baselines.
arXiv:2607. 27737v1 Announce Type: new Abstract: Deep neural networks (DNNs) have achieved remarkable success in classical machine learning problems.
The paper demonstrates that image knowledge distillation can be backdoored even when the teacher model is clean, by poisoning the distillation dataset with triggered and manipulated images that the teacher already classifies as a target label. The attack, effective at poisoning rates as low as 10%, uses targeted adversarial perturbations and GAN-based class transitions to embed a backdoor into the student model while preserving its performance on clean data. The study highlights that the security of knowledge distillation depends not only on the teacher but also on the integrity of the distillation data.
arXiv:2604. 03873v4 Announce Type: replace Abstract: Black-box knowledge distillation for large language models presents a strict trade-off.
The paper introduces IDeaL, a data‑free multi‑teacher distillation technique that generates teacher‑specific, improved samples using decorrelation losses at patch and image levels. By tailoring noise to each teacher, IDeaL produces strong student models that capture complementary teacher information and achieve results close to those distilled from real images. Experiments demonstrate that with only 1,000 images, students trained on IDeaL samples match or exceed the performance of students distilled from a 1,000‑image subset of ImageNet.
The paper introduces Cross-Teacher Guarding for Multi-Teacher On-Policy Distillation (Road MOPD), a method that adds non-routed teachers as auxiliary cross-domain critics to address gradient conflicts in multi-teacher distillation. It employs token-level credit assignment based on teacher agreement and preventive gradient projection to coordinate updates and reduce conflicts. Experiments show that Road MOPD improves performance, especially in high-conflict scenarios, and mitigates degradation in disadvantaged domains.
arXiv:2609.13199v1 Announce Type: new Abstract: Knowledge distillation aims to improve the performance of lightweight student models by transferring knowledge from larger and more powerful teacher mo...
arXiv:2606. 31653v1 Announce Type: cross Abstract: Certified training aims to produce models whose predictions can be formally verified against adversarial perturbations, typically by optimising upper bounds on the worst-case loss over an allowed perturbation set.
The paper introduces Teacher Prediction Refinement Distillation (TPRD), a plug‑in module for Detection Transformers that refines teacher predictions before distillation. TPRD corrects degraded positive predictions and suppresses overconfident negatives, while preserving informative dark knowledge through Maximum Dark Knowledge Preservation. Experiments on MS COCO and PASCAL VOC show that these refinements improve the quality of supervision and the resulting student model’s performance.
arXiv:2609.08798v1 Announce Type: new Abstract: Weak-to-strong generalization asks whether stronger models can learn from weaker supervisors and surpass them. This question is particularly important...
arXiv:2412. 08394v2 Announce Type: replace Abstract: Deep neural networks (DNNs) are vulnerable to adversarial samples crafted by adding imperceptible perturbations to clean data, potentially leading to incorrect and dangerous predictions.
Consistent Distribution Matching is a data‑free distillation technique that accelerates diffusion and flow models by unifying sample generation and score estimation in a single student network. The method requires only a frozen teacher and a trainable student, optimizing a single objective that provably leads to Wasserstein convergence of the student’s flow‑map to the teacher’s marginals. On ImageNet 256×256, it achieves an FID of 2.04 with one function evaluation and 1.37 with four evaluations after just 40 training epochs, outperforming existing data‑free baselines.
The paper introduces Persistent Negative Adversarial Distillation, a method that improves black-box on-policy distillation by maintaining a live pool of historical teacher–student comparisons to stabilize the discriminator’s negative distribution. By anchoring the discriminator with these persistent negatives, the approach reduces reward-estimation error and yields smoother, higher-performing student policies across multiple benchmarks. The study demonstrates that the choice of negative samples is a critical design factor in effective black-box distillation.