Making Knowledge Distillation Cheap Enough to Run at Scale

Open-sourcing Knowledge Distillation Code and Weights of SD-Small and SD-Tiny

KDFP: A first-principles approach to knowledge distillation in large language models

KDFP is a new white‑box methodology for general knowledge distillation in large language models, addressing a gap left by prior work that focused mainly on post‑training abilities such as instruction following and chain‑of‑thought reasoning. The authors evaluate lessons from earlier studies and conduct new experiments to design a distillation approach tailored to modern LLMs. KDFP achieves 1.6%–4.9% higher performance across nine benchmarks and improves training efficiency by up to 99.1% through reduced temporary parameters.

Is Knowledge Distillation Actually Greener? A Case Study in Machine Translation

arXiv:2602.09691v2 Announce Type: replace-cross Abstract: Knowledge distillation (KD) is a technique to compress a larger teacher system into a smaller student. In machine translation, KD is commonly...

Revisiting the Capacity Gap in Chain-of-Thought Distillation from a Practical Perspective

arXiv:2604.08880v2 Announce Type: replace-cross Abstract: Chain-of-thought (CoT) distillation transfers reasoning behaviors from a strong teacher to a smaller student, but prior work reports a capaci...

Model Distillation in the API

Fine-tune a cost-efficient model with the outputs of a large frontier model–all on the OpenAI platform

Recursive Offline Distillation via Entropy Gain Sampling and Answer Mixing

arXiv:2610.20914v1 Announce Type: new Abstract: Distillation of reasoning chains from Large Language Models (LLMs) into their smaller siblings is a standard training approach. Frequently, researchers...