Open-sourcing Knowledge Distillation Code and Weights of SD-Small and SD-Tiny

Making Knowledge Distillation Cheap Enough to Run at Scale

Apertus LLM Family Expansion via Distillation and Quantization

arXiv:2605. 29128v2 Announce Type: replace Abstract: The wide adoption of LLMs has led to their use in great variety of applications and scenarios, such as chatbot assistants and data annotation, creating the need for the models to satisfy certain budget and hardware constraints.

Model Distillation in the API

Fine-tune a cost-efficient model with the outputs of a large frontier model–all on the OpenAI platform

KDFlow: A User-Friendly and Efficient Knowledge Distillation Framework for Large Language Models

arXiv:2603. 01875v3 Announce Type: replace-cross Abstract: Knowledge distillation (KD) is an essential technique to compress large language models (LLMs) into smaller ones.

Distill

We’re excited to support today’s launch of Distill, a new kind of journal aimed at excellent communication of machine learning results (novel or existing).

Efficient Knowledge Distillation for LLMs: Offline Top-K Logits and a Fused Chunked KL Loss

arXiv:2608. 03796v1 Announce Type: cross Abstract: Small language models are often the only option for deployment under tight latency, cost, and on-premises constraints, but they are rarely trained from scratch: a compressed model is usually recovered through knowledge distillation (KD).

Characterize Then Distill: Mechanistic Reasoning in Large Output Spaces

arXiv:2606. 06840v1 Announce Type: cross Abstract: Modern reasoning models offer surprisingly strong zero-shot performance on challenging multi-label tasks that require selecting a small set of relevant options from hundreds of thousands to millions of candidate labels.

IDeaL: Data-Free Multi-Teacher Distillation via Improved Dead Leaves

The paper introduces IDeaL, a data‑free multi‑teacher distillation technique that generates teacher‑specific, improved samples using decorrelation losses at patch and image levels. By tailoring noise to each teacher, IDeaL produces strong student models that capture complementary teacher information and achieve results close to those distilled from real images. Experiments demonstrate that with only 1,000 images, students trained on IDeaL samples match or exceed the performance of students distilled from a 1,000‑image subset of ImageNet.

SelFusion: Self-distillation for Diffusion Language Models

arXiv:2608.22898v1 Announce Type: new Abstract: Diffusion language models (DLMs) alleviate the inherent latency bottleneck of autoregressive (AR) large language models (LLMs), but their degraded gene...

A Controlled Study of Feature-Based Knowledge Distillation Across Student Designs

arXiv:2608. 08294v1 Announce Type: new Abstract: Knowledge distillation trains a smaller student to match the outputs of a larger teacher.

Beyond FLOPs: Energy-Aware Knowledge Distillation for Sustainable LLMs on Code-Related Task

The paper explores energy-aware knowledge distillation for large language models (LLMs) used in software engineering tasks such as clone detection, vulnerability prediction, and code summarization. It shows that the commonly used FLOPs metric does not reliably reflect actual energy consumption, and that using energy-surrogate models during distillation can reduce inference energy by up to 90% and memory usage by 86% with only modest accuracy loss. The study demonstrates that guiding distillation with direct energy estimates improves the sustainability and deployability of LLMs on consumer hardware.

Optimizing Teacher-Student Partitioning for Scalable Knowledge Distillation on HPC Systems

arXiv:2606. 27797v1 Announce Type: cross Abstract: Knowledge Distillation (KD) enables training smaller student models under the guidance of larger teacher models, and the widely adopted TRL library implements it.