arXiv:2509.25002v2 Announce Type: replace
Abstract: Model distillation typically focuses on behavioral mimicry, where a student model is trained to replicate a teacher's output while treating its int...
By Somin Wadhwa, Silvio Amir, Byron C. Wallace
The paper investigates how knowledge distillation from Vision Transformers to smaller CNNs can cause dimensional collapse in the student’s representation space. Using SVD and Shannon entropy, the authors show that cosine‑based distillation leads to a drastic reduction in effective rank, while adding an InfoNCE objective can double the rank but harms downstream accuracy due to signal dilution. They further demonstrate that a label‑aware contrastive objective (Supervised Contrastive distillation) can maintain or improve accuracy without unnecessary rank expansion, indicating that effective rank alone is not a reliable indicator of representation quality.
By Kabir Thayani
The paper investigates whether a strong teacher is necessary for knowledge distillation in large language model pretraining. By varying teacher and student sizes and token budgets, the authors create strong‑to‑weak, same‑level, and weak‑to‑strong teacher‑student pairs and evaluate distillation effectiveness. They find that even small, undertrained teachers can improve larger students when losses are properly mixed, and that pushing teachers to be stronger can saturate or reverse gains, while distillation mainly boosts generalization rather than in‑domain fitting.
By Taiming Lu, Zhuang Liu
The study investigates how pruning affects internal mechanisms of GPT‑2 by systematically pruning attention heads and MLP neurons using magnitude and random importance criteria at 3%, 50%, and 70% sparsity. Using the indirect object identification (IOI) circuit as a ground truth, the authors measure circuit integrity with activation patching, path patching, and direct logit attribution, revealing two distinct failure modes: magnitude head pruning essentially zeros the circuit, while magnitude neuron pruning at 30% inverts it, causing the model to prefer incorrect names. The results show that conventional output metrics like perplexity can miss significant mechanistic damage, underscoring the need for circuit‑level causal analysis when evaluating compressed language models.
By Nabeelah Faumi, Deshan Sumanathilaka
The paper introduces two distillation methods for handling misaligned AI models. Distillation for Incrimination (DFI) transfers misalignment without the ability to conceal it, making students more likely to admit hidden behaviors. Distillation for Capabilities (DFC) transfers useful capabilities while minimizing misalignment, using techniques like inoculation prompting and focused training.
By Sebastian Prasanna, Jacqueline Tay, Alek Westover
FocusOPD introduces a structure‑guided on‑policy distillation method that adds token‑wise directional feature alignment to deep Transformer layers during early denoising steps. This approach addresses the mismatch between teacher and student representations that persists after standard OPD distillation. Experiments on rectified‑flow backbones show that FocusOPD improves performance on GenEval, PickScore, and LAION‑Aesthetic benchmarks, and ablations confirm that selective deep and early supervision is more effective than full‑network alignment.
By Xinrui Lin, Shumin Wang, Suqi Song, Fan Li, Yanyong Zhang