arXiv AI June 2, 2026 By Wendao Wu, Fangqing Zhang, Haihan Zhang, Cong Fang

What Makes a Strong Model? A Unified Spectral Analysis of Knowledge Transfer over High-dimensional Linear Regression

arXiv:2606. 01292v1 Announce Type: cross Abstract: Teacher-Student Knowledge Transfer (KT) is ubiquitous in modern machine learning, ranging from classical model compression via Knowledge Distillation (KD) to the emergent phenomenon of Weak-to-Strong (W2S) generalization.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

efficiency

arXiv Machine Learning

Jun 25

Knowledge Cascade: Reverse Knowledge Distillation on Nonparametric Multivariate Functional Estimation

arXiv:2606. 25927v1 Announce Type: cross Abstract: As machine learning models and datasets continue to grow, developing complex models has become increasingly computationally demanding.

By Luyang Fang, Haoran Lu, Yongkai Chen, Wenxuan Zhong, Ping Ma

efficiency

arXiv Machine Learning

Jul 9

Fixed-Gaussian Spectral Algorithms: Minimax Optimal Rates for Misspecified Learning and Transfer

arXiv:2501. 10870v2 Announce Type: replace-cross Abstract: The principal objective of this work is twofold within nonparametric regression settings: (1) to establish the minimax optimal convergence rates for fixed-bandwidth Gaussian kernel spectral algorithms when the true regression function resides in a Sobolev space, and (2) to apply Gaussian spectral algorithms for achieving robust and adaptive transfer learning under concept shift.

By Haotian Lin, Matthew Reimherr

fine-tuning

arXiv Machine Learning

Jul 8

SGD-Based Knowledge Distillation with Bayesian Teachers: Theory and Guidelines

arXiv:2601. 01484v2 Announce Type: replace Abstract: Knowledge Distillation (KD) is a central paradigm for transferring knowledge from a large teacher network to a typically smaller student model, often by leveraging soft probabilistic outputs.

By Itai Morad, Nir Shlezinger, Yonina C. Eldar

efficiency

arXiv Machine Learning

Jul 28

SPRKD: Effective Knowledge Distillation for Deep Neural Networks via Saddle Region Approximation

arXiv:2607. 23346v1 Announce Type: new Abstract: Modern deep neural networks are potent catalysts for scientific and industrial impact, yet excessive parameter counts impede deployment in low-compute settings such as hospital equipment and energy infrastructure.

By Aditya Dewan, Arjun Yogeswaran, Benjamin Fedoruk

rag fine-tuning efficiency multimodal benchmarks

arXiv Machine Learning

Sep 22

Do Student LLMs Inherit OOD Robustness? Invariance-Weighted Distillation for Reliable Knowledge Transfer

arXiv:2609.22566v1 Announce Type: cross Abstract: Knowledge distillation (KD) aims to compress high-performance teacher LLMs into lightweight students. However, distilled students often exhibit subst...

By Dileesha Kannangara, Sanghamitra Dutta

llms nlp efficiency benchmarks

arXiv Machine Learning

Sep 15

Discovering and Preserving Category Correlation Knowledge via Adaptive Reciprocal Knowledge Distillation

arXiv:2609.13199v1 Announce Type: new Abstract: Knowledge distillation aims to improve the performance of lightweight student models by transferring knowledge from larger and more powerful teacher mo...

By Dawen Jiang, Zhishu Shen, Zeyu Liu, Tiehua Zhang

efficiency benchmarks safety

One email a morning, machine-written

Related stories

Knowledge Cascade: Reverse Knowledge Distillation on Nonparametric Multivariate Functional Estimation

Fixed-Gaussian Spectral Algorithms: Minimax Optimal Rates for Misspecified Learning and Transfer

SGD-Based Knowledge Distillation with Bayesian Teachers: Theory and Guidelines

SPRKD: Effective Knowledge Distillation for Deep Neural Networks via Saddle Region Approximation

Do Student LLMs Inherit OOD Robustness? Invariance-Weighted Distillation for Reliable Knowledge Transfer

Discovering and Preserving Category Correlation Knowledge via Adaptive Reciprocal Knowledge Distillation