arXiv Computation and Language
Circuit Distillation
Read the original on arXiv Computation and Language →The Flow has not summarised this story yet — read it at arXiv Computation and Language.
Related stories
arXiv Computation and Language
Oct 28
What Survives Distillation? A Mechanistic Interpretability Study of GPT-2 and DistilGPT-2
The paper investigates how knowledge distillation affects the internal mechanisms of language models by comparing GPT‑2 with its six‑layer student, DistilGPT‑2. Using techniques such as linear CKA, logit lens analysis, and attention pattern decomposition, the authors examine two circuits: Indirect Object Identification (IOI) and Greater‑Than (GT). They find that the IOI circuit collapses completely in the distilled model, whereas the GT circuit largely survives, with the student’s layer‑4 head effectively remapping the teacher’s layer‑9 comparison head, achieving 87.7% of the original task performance.
arXiv Machine Learning
Oct 9
KDFP: A first-principles approach to knowledge distillation in large language models
KDFP is a new white‑box methodology for general knowledge distillation in large language models, addressing a gap left by prior work that focused mainly on post‑training abilities such as instruction following and chain‑of‑thought reasoning. The authors evaluate lessons from earlier studies and conduct new experiments to design a distillation approach tailored to modern LLMs. KDFP achieves 1.6%–4.9% higher performance across nine benchmarks and improves training efficiency by up to 99.1% through reduced temporary parameters.
arXiv AI
1d ago
UniSD: Towards a Unified Self-Distillation Framework for Large Language Models
arXiv:2605.06597v3 Announce Type: replace-cross Abstract: Self-distillation (SD) offers a promising path for adapting large language models (LLMs) without relying on stronger external teachers. Howev...
arXiv AI
Oct 15
OPMD: On-Policy Mechanism Distillation Beyond Output Supervision
arXiv:2610.14022v1 Announce Type: cross Abstract: On-policy distillation (OPD) improves language-model by supervising a student on prefixes sampled from its own policy, but its supervision remains co...
arXiv Computation and Language
Oct 28
Bridged Self-Distillation for On-Policy Learning
Bridged Self-Distillation (BSD) is a unified framework that enables on‑policy distillation from any teacher providing reasoning traces. The external teacher supplies a privileged reasoning trace offline, while a self‑teacher—an identical copy of the student conditioned on that trace—scores the student’s rollouts in its own token space. BSD uses these token‑level signals, along with a verifier, to adjust updates on unsuccessful rollouts, allowing the same training procedure to work with shared‑tokenizer, cross‑tokenizer, and closed‑API teachers and consistently outperforming scenario‑specific baselines.
arXiv Machine Learning
Jul 14
Reference-Based Distillation Detection in LLMs
arXiv:2607. 09692v1 Announce Type: new Abstract: Model distillation -- training on outputs from stronger third-party models -- is widely used to boost performance, but raises concerns about unfair advantages and policy violations.