arXiv Machine Learning
Recursive Offline Distillation via Entropy Gain Sampling and Answer Mixing
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Oct 9
KDFP: A first-principles approach to knowledge distillation in large language models
KDFP is a new white‑box methodology for general knowledge distillation in large language models, addressing a gap left by prior work that focused mainly on post‑training abilities such as instruction following and chain‑of‑thought reasoning. The authors evaluate lessons from earlier studies and conduct new experiments to design a distillation approach tailored to modern LLMs. KDFP achieves 1.6%–4.9% higher performance across nine benchmarks and improves training efficiency by up to 99.1% through reduced temporary parameters.
arXiv Machine Learning
Sep 30
Know Thyself, Teach Thyself: Internal Information Flow for Selective Self-Distillation
arXiv:2609.36695v1 Announce Type: new Abstract: Self-distillation turns knowledge distillation into a closed learning loop and offers a path toward recursive self-improvement. Without an external tea...
arXiv Machine Learning
Jun 9
Breaking the Tokenizer Barrier: On-Policy Distillation across Model Families
arXiv:2606. 09456v1 Announce Type: new Abstract: On-Policy Distillation (OPD) has become a core technique in the post-training of Large Language Models (LLMs) for transferring knowledge from domain experts to student models.
arXiv AI
Sep 7
What Matters in On-Policy Distillation? A Perspective on Data Efficiency and Data Selection
The paper investigates data efficiency and selection in On‑Policy Distillation (OPD) for large language models. It shows that 1‑shot OPD—training on a single example—consistently improves performance, especially when the example is hard, and that longer chain‑of‑thought (CoT) paths drive the gains rather than token entropy. Based on these findings, the authors propose a simple hard‑example selection strategy that, using only eight carefully chosen hard examples, matches the performance of a 17,000‑example baseline across models from 1.5B to 7B parameters.
arXiv Machine Learning
Oct 8
Collaborative Reasoning Distillation via Cross-Feedback and Coherent Curation
arXiv:2610.09587v1 Announce Type: new Abstract: Reasoning capabilities are critical for advancing Large Language Models, yet current approaches either require massive computational budgets or struggl...
arXiv AI
Jul 1
Distilling the Essence: Efficient Reasoning Distillation via Sequence Truncation
arXiv:2512. 21002v3 Announce Type: replace-cross Abstract: Distilling the capabilities from a large reasoning model (LRM) to a smaller student model often involves training on substantial amounts of reasoning data.