Open-sourcing Knowledge Distillation Code and Weights of SD-Small and SD-Tiny

Making Knowledge Distillation Cheap Enough to Run at Scale

KDFP: A first-principles approach to knowledge distillation in large language models

KDFP is a new white‑box methodology for general knowledge distillation in large language models, addressing a gap left by prior work that focused mainly on post‑training abilities such as instruction following and chain‑of‑thought reasoning. The authors evaluate lessons from earlier studies and conduct new experiments to design a distillation approach tailored to modern LLMs. KDFP achieves 1.6%–4.9% higher performance across nine benchmarks and improves training efficiency by up to 99.1% through reduced temporary parameters.

Apertus LLM Family Expansion via Distillation and Quantization

arXiv:2605. 29128v2 Announce Type: replace Abstract: The wide adoption of LLMs has led to their use in great variety of applications and scenarios, such as chatbot assistants and data annotation, creating the need for the models to satisfy certain budget and hardware constraints.

Model Distillation in the API

Fine-tune a cost-efficient model with the outputs of a large frontier model–all on the OpenAI platform

Is Knowledge Distillation Actually Greener? A Case Study in Machine Translation

arXiv:2602.09691v2 Announce Type: replace-cross Abstract: Knowledge distillation (KD) is a technique to compress a larger teacher system into a smaller student. In machine translation, KD is commonly...

Revisiting the Capacity Gap in Chain-of-Thought Distillation from a Practical Perspective

arXiv:2604.08880v2 Announce Type: replace-cross Abstract: Chain-of-thought (CoT) distillation transfers reasoning behaviors from a strong teacher to a smaller student, but prior work reports a capaci...