arXiv Machine Learning
Beyond Imitation: Auditing the Recoverability of Reasoning in Distilled Models
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Jul 28
Outcome-Confounded Local Supervision in On-Policy Distillation
arXiv:2607. 23731v1 Announce Type: new Abstract: On-policy distillation (OPD) trains a student on its own trajectories while a teacher supplies dense token-level likelihoods at student-visited prefixes.
arXiv AI
Sep 17
Trajectory Learnability for Offline On-Policy Distillation with Imperfect Teachers
The paper introduces a method for offline on‑policy distillation that addresses the problem of imperfect teacher supervision. By training on teacher‑successful problems and measuring changes in token likelihoods on teacher‑failed trajectories, the authors derive a learnability signal that weights the distillation loss. This approach improves performance on mathematical reasoning and code generation tasks while reducing computational cost compared to online distillation.
arXiv Computation and Language
4d ago
Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models
arXiv:2609.36734v1 Announce Type: new Abstract: Knowledge Distillation (KD) trains a smaller-capacity student model to imitate a larger-capacity teacher model by matching output distributions, implic...
Hugging Face Trending Papers
Sep 16
Trajectory Learnability for Offline On-Policy Distillation with Imperfect Teachers
Offline on-policy distillation gains efficiency by collecting student trajectories and teacher supervision once and reusing them throughout optimization. The same reuse makes imperfect supervision per...
arXiv Machine Learning
Aug 18
Rethinking Reverse KL as Adaptive Entropy Distillation
arXiv:2608. 14685v1 Announce Type: new Abstract: Knowledge distillation (KD) is widely used to transfer the capabilities of large language models (LLMs) to smaller students, but existing objectives often struggle to balance faithful imitation and robust generation.
arXiv Machine Learning
Jun 16
Entropy-Aware On-Policy Distillation of Language Models
arXiv:2603. 07079v3 Announce Type: replace Abstract: On-policy distillation is a promising approach for transferring knowledge between language models, where a student learns from dense token-level signals along its own trajectories.