arXiv Machine Learning
Know Thyself, Teach Thyself: Internal Information Flow for Selective Self-Distillation
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Computation and Language
1d ago
Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models
arXiv:2609.36734v1 Announce Type: new Abstract: Knowledge Distillation (KD) trains a smaller-capacity student model to imitate a larger-capacity teacher model by matching output distributions, implic...
arXiv Computation and Language
Sep 2
Knowledge Distillation During Mid-Training Favors Reasoning over Factual Recall
arXiv:2609.01532v1 Announce Type: new Abstract: Logit-based knowledge distillation (KD) is used to train smaller language models (LMs) via supervision from stronger teachers, but whether its benefits...
arXiv Machine Learning
Jul 7
Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipe
arXiv:2605. 03677v2 Announce Type: replace Abstract: On-policy distillation (OPD) has recently emerged as an effective post-training paradigm for consolidating the capabilities of specialized expert models into a single student model.
arXiv Machine Learning
Aug 7
On-Policy Self-Distillation without Any Supervision
arXiv:2608. 06296v1 Announce Type: new Abstract: On-policy (Self-)Distillation (OPD / OPSD) has shown strong potential for post-training large language models (LLMs).
arXiv AI
Jul 3
DemoPSD: Disagreement-Modulated Policy Self-Distillation
arXiv:2607. 02502v1 Announce Type: cross Abstract: On-policy self-distillation (OPSD) has emerged as a practical method for training large language models (LLMs) to reason, where a single model acts as both the teacher and the student with different levels of information access.
arXiv AI
Jul 21
Distilled Reinforcement Learning for LLM Post-training
arXiv:2607. 17247v1 Announce Type: cross Abstract: Large language model (LLM) post-training is essential for improving reasoning, adaptation, and alignment.