arXiv Machine Learning
iSDFT: Information-Proximal Self-Distillation for Continual Learning in LLMs
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
Hugging Face Trending Papers
Jul 2
Denser $\neq$ Better: Limits of On-Policy Self-Distillation for Continual Post-Training
Continual post-training enables foundation models to acquire new knowledge while preserving existing capabilities. Recent work suggests that on-policy learning can mitigate forgetting, with on-policy self-distillation emerging as a particularly attractive approach.
arXiv Machine Learning
Jul 3
Denser $\neq$ Better: Limits of On-Policy Self-Distillation for Continual Post-Training
arXiv:2607. 01763v1 Announce Type: new Abstract: Continual post-training enables foundation models to acquire new knowledge while preserving existing capabilities.
arXiv Machine Learning
Aug 10
Self-Distillation Enables Continual Learning
arXiv:2601. 19897v2 Announce Type: replace Abstract: Continual learning, enabling models to acquire new skills and knowledge without degrading existing capabilities, remains a fundamental challenge for foundation models.
arXiv AI
Jul 3
DemoPSD: Disagreement-Modulated Policy Self-Distillation
arXiv:2607. 02502v1 Announce Type: cross Abstract: On-policy self-distillation (OPSD) has emerged as a practical method for training large language models (LLMs) to reason, where a single model acts as both the teacher and the student with different levels of information access.
arXiv AI
Jun 10
Adaptive Teacher Exposure for Self-Distillation in LLM Reasoning
arXiv:2605. 11458v3 Announce Type: replace Abstract: On-policy self-distillation has become a strong recipe for LLM reasoning, where a privileged teacher supervises the student's own rollouts while conditioning on the reference solution.
arXiv Computation and Language
6d ago
Recursive Self-Improvement via On-Policy Distillation for Reasoning
The paper introduces a recursive self-improvement framework for language models that replaces an external teacher with a frozen copy of the student, enabling dynamic co-evolution (DCE) and self-refined concise learning (SRCL). DCE allows the privileged teacher to evolve alongside the student, while SRCL trains on shorter, verified rewrites to reduce verbosity. Experiments show that the combined DCE+SRCL approach outperforms traditional on‑policy self‑distillation across multiple model sizes and math benchmarks, achieving significant accuracy gains and shorter outputs.