arXiv Machine Learning June 25, 2026 By Luyang Fang, Haoran Lu, Yongkai Chen, Wenxuan Zhong, Ping Ma

Knowledge Cascade: Reverse Knowledge Distillation on Nonparametric Multivariate Functional Estimation

Read the original on arXiv Machine Learning →

arXiv:2606. 25927v1 Announce Type: cross Abstract: As machine learning models and datasets continue to grow, developing complex models has become increasingly computationally demanding.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

efficiency

arXiv AI

Jun 2

What Makes a Strong Model? A Unified Spectral Analysis of Knowledge Transfer over High-dimensional Linear Regression

arXiv:2606. 01292v1 Announce Type: cross Abstract: Teacher-Student Knowledge Transfer (KT) is ubiquitous in modern machine learning, ranging from classical model compression via Knowledge Distillation (KD) to the emergent phenomenon of Weak-to-Strong (W2S) generalization.

By Wendao Wu, Fangqing Zhang, Haihan Zhang, Cong Fang

efficiency

arXiv Machine Learning

Jul 21

Representation-Aware Distributionally Robust Optimization: A Knowledge Transfer Framework

arXiv:2509. 09371v2 Announce Type: replace-cross Abstract: Distributionally robust optimization (DRO) protects statistical learning against distributional shifts by optimizing the worst-case performance over a set of perturbed distributions.

By Zitao Wang, Nian Si, Molei Liu

fine-tuning

arXiv Machine Learning

Jun 25

Distill on a Diet: Efficient Knowledge Distillation via Learnable Data Pruning

arXiv:2606. 25488v1 Announce Type: new Abstract: Knowledge Distillation (KD) is widely used to obtain compact models for efficient inference in resource-constrained environments.

By Yifan Wu, Yiqi Wang, Xichen Ye, Wenjing Yan, Xiaoqiang Li, Cheng Jin, Xiangyu Yue, Weizhong Zhang

efficiency

Hugging Face Trending Papers

Jun 22

Understanding Knowledge Distillation in Post-Training: When It Helps and When It Fails

Large language models (LLMs) achieve strong performance across many tasks, but their high computational cost limits deployment in resource-constrained environments. Knowledge Distillation (KD) offers a practical solution by transferring knowledge from a teacher model of a larger size to a smaller student model.

llms fine-tuning efficiency

arXiv Machine Learning

Jul 28

SPRKD: Effective Knowledge Distillation for Deep Neural Networks via Saddle Region Approximation

arXiv:2607. 23346v1 Announce Type: new Abstract: Modern deep neural networks are potent catalysts for scientific and industrial impact, yet excessive parameter counts impede deployment in low-compute settings such as hospital equipment and energy infrastructure.

By Aditya Dewan, Arjun Yogeswaran, Benjamin Fedoruk

rag fine-tuning efficiency multimodal benchmarks

arXiv Machine Learning

Sep 22

Do Student LLMs Inherit OOD Robustness? Invariance-Weighted Distillation for Reliable Knowledge Transfer

arXiv:2609.22566v1 Announce Type: cross Abstract: Knowledge distillation (KD) aims to compress high-performance teacher LLMs into lightweight students. However, distilled students often exhibit subst...

By Dileesha Kannangara, Sanghamitra Dutta

llms nlp efficiency benchmarks

One email a morning, machine-written

Related stories

What Makes a Strong Model? A Unified Spectral Analysis of Knowledge Transfer over High-dimensional Linear Regression

Representation-Aware Distributionally Robust Optimization: A Knowledge Transfer Framework

Distill on a Diet: Efficient Knowledge Distillation via Learnable Data Pruning

Understanding Knowledge Distillation in Post-Training: When It Helps and When It Fails

SPRKD: Effective Knowledge Distillation for Deep Neural Networks via Saddle Region Approximation

Do Student LLMs Inherit OOD Robustness? Invariance-Weighted Distillation for Reliable Knowledge Transfer