The paper investigates how knowledge distillation affects the internal mechanisms of language models by comparing GPT‑2 with its six‑layer student, DistilGPT‑2. Using techniques such as linear CKA, logit lens analysis, and attention pattern decomposition, the authors examine two circuits: Indirect Object Identification (IOI) and Greater‑Than (GT). They find that the IOI circuit collapses completely in the distilled model, whereas the GT circuit largely survives, with the student’s layer‑4 head effectively remapping the teacher’s layer‑9 comparison head, achieving 87.7% of the original task performance.
arXiv:2509.25002v2 Announce Type: replace Abstract: Model distillation typically focuses on behavioral mimicry, where a student model is trained to replicate a teacher's output while treating its int...
The paper investigates how knowledge distillation from Vision Transformers to smaller CNNs can cause dimensional collapse in the student’s representation space. Using SVD and Shannon entropy, the authors show that cosine‑based distillation leads to a drastic reduction in effective rank, while adding an InfoNCE objective can double the rank but harms downstream accuracy due to signal dilution. They further demonstrate that a label‑aware contrastive objective (Supervised Contrastive distillation) can maintain or improve accuracy without unnecessary rank expansion, indicating that effective rank alone is not a reliable indicator of representation quality.
The paper investigates whether a strong teacher is necessary for knowledge distillation in large language model pretraining. By varying teacher and student sizes and token budgets, the authors create strong‑to‑weak, same‑level, and weak‑to‑strong teacher‑student pairs and evaluate distillation effectiveness. They find that even small, undertrained teachers can improve larger students when losses are properly mixed, and that pushing teachers to be stronger can saturate or reverse gains, while distillation mainly boosts generalization rather than in‑domain fitting.
The study investigates how pruning affects internal mechanisms of GPT‑2 by systematically pruning attention heads and MLP neurons using magnitude and random importance criteria at 3%, 50%, and 70% sparsity. Using the indirect object identification (IOI) circuit as a ground truth, the authors measure circuit integrity with activation patching, path patching, and direct logit attribution, revealing two distinct failure modes: magnitude head pruning essentially zeros the circuit, while magnitude neuron pruning at 30% inverts it, causing the model to prefer incorrect names. The results show that conventional output metrics like perplexity can miss significant mechanistic damage, underscoring the need for circuit‑level causal analysis when evaluating compressed language models.
The paper introduces two distillation methods for handling misaligned AI models. Distillation for Incrimination (DFI) transfers misalignment without the ability to conceal it, making students more likely to admit hidden behaviors. Distillation for Capabilities (DFC) transfers useful capabilities while minimizing misalignment, using techniques like inoculation prompting and focused training.
FocusOPD introduces a structure‑guided on‑policy distillation method that adds token‑wise directional feature alignment to deep Transformer layers during early denoising steps. This approach addresses the mismatch between teacher and student representations that persists after standard OPD distillation. Experiments on rectified‑flow backbones show that FocusOPD improves performance on GenEval, PickScore, and LAION‑Aesthetic benchmarks, and ablations confirm that selective deep and early supervision is more effective than full‑network alignment.
arXiv:2605. 31191v2 Announce Type: replace Abstract: We investigate how teacher-student capacity relationships modulate knowledge distillation (KD) effectiveness in ResNet-based image classification on CIFAR-10.
arXiv:2603. 26556v2 Announce Type: replace-cross Abstract: Converting a pretrained Transformer into a more efficient hybrid model through distillation offers a promising approach to reducing inference costs.
The paper introduces LastOPD, a method that mitigates collapse in latent on‑policy distillation by applying latent supervision only to the last‑layer state during a brief cross‑fade into token‑level OPD. Experiments show that LastOPD improves MATH‑500 accuracy by 5.55 and 4.02 points over token‑only OPD when distilling Qwen3‑4B and Qwen3‑8B into Qwen3‑1.7B‑Base, and achieves comparable final scores in roughly half the training steps.
arXiv:2608. 03796v1 Announce Type: cross Abstract: Small language models are often the only option for deployment under tight latency, cost, and on-premises constraints, but they are rarely trained from scratch: a compressed model is usually recovered through knowledge distillation (KD).
arXiv:2407. 13911v5 Announce Type: replace-cross Abstract: Prompt-based continual learning has shown strong performance in rehearsal-free class-incremental learning by adapting learnable prompts while freezing a pre-trained Vision Transformer (ViT) backbone.
arXiv:2609.22566v1 Announce Type: cross Abstract: Knowledge distillation (KD) aims to compress high-performance teacher LLMs into lightweight students. However, distilled students often exhibit subst...