Hugging Face Trending Papers July 22, 2026

When Does Knowledge Distillation Hurt? Reliability-Aware Distillation for Low-Resource Language Summarization

Read the original on Hugging Face Trending Papers →

Knowledge distillation (KD) is a standard approach for compressing sequence-to-sequence models, but its per-sample effects are rarely examined. On the BanSum Bangla summarization benchmark, we find that standard KD improves ROUGE-L by only +0.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at Hugging Face Trending Papers.

arXiv AI

Jul 23

When Does Knowledge Distillation Hurt? Reliability-Aware Distillation for Low-Resource Language Summarization

arXiv:2607. 19956v1 Announce Type: cross Abstract: Knowledge distillation (KD) is a standard approach for compressing sequence-to-sequence models, but its per-sample effects are rarely examined.

By Dipto Sumit, Ankan Kumar Roy Srizon, Sadia Khair Rodela, Atia Haque Asha, Mourchona Afrin, Niloy Farhan, Farig Sadeque

nlp fine-tuning efficiency benchmarks safety

arXiv Computation and Language

4d ago

Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models

arXiv:2609.36734v1 Announce Type: new Abstract: Knowledge Distillation (KD) trains a smaller-capacity student model to imitate a larger-capacity teacher model by matching output distributions, implic...

By Ayan Sengupta, Vaibhav Seth, Tanmoy Chakraborty

llms efficiency benchmarks safety

arXiv Statistics ML

2d ago

Unbiased Top-$k$ Estimation for On-Policy Distillation

arXiv:2609.34447v2 Announce Type: replace-cross Abstract: On-policy distillation (OPD) is becoming an important component of large language model (LLM) post-training for transferring the reasoning ca...

By Linjian Meng, Siyuan Gan, YuHan Li, Xiran Wang, Ziyang Ding, Ditang Gou, Yiming Wu, Zhen Zhao

llms efficiency safety

arXiv Machine Learning

Sep 22

Do Student LLMs Inherit OOD Robustness? Invariance-Weighted Distillation for Reliable Knowledge Transfer

arXiv:2609.22566v1 Announce Type: cross Abstract: Knowledge distillation (KD) aims to compress high-performance teacher LLMs into lightweight students. However, distilled students often exhibit subst...

By Dileesha Kannangara, Sanghamitra Dutta

llms nlp efficiency benchmarks

arXiv Computation and Language

Sep 2

Knowledge Distillation During Mid-Training Favors Reasoning over Factual Recall

arXiv:2609.01532v1 Announce Type: new Abstract: Logit-based knowledge distillation (KD) is used to train smaller language models (LMs) via supervision from stronger teachers, but whether its benefits...

By Jacqueline He, Howard Yen, Shuyue Stella Li, Margaret Li, Hanqing Zeng, Yinglong Xia, Benyu Zhang, Zhuokai Zhao, Qiang Zhang, Pang Wei Koh, Luke Zettlemoyer, Wen-tau Yih

llms efficiency

arXiv AI

Jul 20

When Perplexity Lies: Generation-Focused Distillation of Hybrid Sequence Models

arXiv:2603. 26556v2 Announce Type: replace-cross Abstract: Converting a pretrained Transformer into a more efficient hybrid model through distillation offers a promising approach to reducing inference costs.

By Juan Gabriel Kostelec, Qinghai Guo

llms efficiency benchmarks

One email a morning, machine-written

Related stories

When Does Knowledge Distillation Hurt? Reliability-Aware Distillation for Low-Resource Language Summarization

Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models

Unbiased Top-$k$ Estimation for On-Policy Distillation

Do Student LLMs Inherit OOD Robustness? Invariance-Weighted Distillation for Reliable Knowledge Transfer

Knowledge Distillation During Mid-Training Favors Reasoning over Factual Recall

When Perplexity Lies: Generation-Focused Distillation of Hybrid Sequence Models