arXiv Machine Learning
Is Knowledge Distillation Actually Greener? A Case Study in Machine Translation
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv AI
Jul 20
KDFlow: A User-Friendly and Efficient Knowledge Distillation Framework for Large Language Models
arXiv:2603. 01875v3 Announce Type: replace-cross Abstract: Knowledge distillation (KD) is an essential technique to compress large language models (LLMs) into smaller ones.
Hugging Face Trending Papers
Jun 22
Understanding Knowledge Distillation in Post-Training: When It Helps and When It Fails
Large language models (LLMs) achieve strong performance across many tasks, but their high computational cost limits deployment in resource-constrained environments. Knowledge Distillation (KD) offers a practical solution by transferring knowledge from a teacher model of a larger size to a smaller student model.
arXiv AI
Aug 5
TQLite: Multi-LLM Jury Guided Distillation for Real-time MQM Translation Quality Evaluation
arXiv:2608. 02975v1 Announce Type: cross Abstract: Large language models (LLMs) have demonstrated impressive performance in MQM-based translation quality (TQ) evaluation, and recent advances in large reasoning models (LRMs) promise even greater improvements.
arXiv Computation and Language
Aug 25
SelFusion: Self-distillation for Diffusion Language Models
arXiv:2608.22898v1 Announce Type: new Abstract: Diffusion language models (DLMs) alleviate the inherent latency bottleneck of autoregressive (AR) large language models (LLMs), but their degraded gene...
arXiv Machine Learning
4d ago
REVO: Rollout-Efficient Off-Policy Distillation via Variance-Guided Reuse
arXiv:2609.37500v1 Announce Type: new Abstract: On-policy distillation (OPD) trains language models using dense token-level teacher supervision on student-generated trajectories. However, its relianc...
arXiv AI
Aug 5
Efficient Knowledge Distillation for LLMs: Offline Top-K Logits and a Fused Chunked KL Loss
arXiv:2608. 03796v1 Announce Type: cross Abstract: Small language models are often the only option for deployment under tight latency, cost, and on-premises constraints, but they are rarely trained from scratch: a compressed model is usually recovered through knowledge distillation (KD).