arXiv Machine Learning July 8, 2026 By Itai Morad, Nir Shlezinger, Yonina C. Eldar

SGD-Based Knowledge Distillation with Bayesian Teachers: Theory and Guidelines

Read the original on arXiv Machine Learning →

arXiv:2601. 01484v2 Announce Type: replace Abstract: Knowledge Distillation (KD) is a central paradigm for transferring knowledge from a large teacher network to a typically smaller student model, often by leveraging soft probabilistic outputs.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

efficiency

arXiv Computation and Language

4d ago

Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models

arXiv:2609.36734v1 Announce Type: new Abstract: Knowledge Distillation (KD) trains a smaller-capacity student model to imitate a larger-capacity teacher model by matching output distributions, implic...

By Ayan Sengupta, Vaibhav Seth, Tanmoy Chakraborty

llms efficiency benchmarks safety

arXiv Machine Learning

Sep 15

Discovering and Preserving Category Correlation Knowledge via Adaptive Reciprocal Knowledge Distillation

arXiv:2609.13199v1 Announce Type: new Abstract: Knowledge distillation aims to improve the performance of lightweight student models by transferring knowledge from larger and more powerful teacher mo...

By Dawen Jiang, Zhishu Shen, Zeyu Liu, Tiehua Zhang

efficiency benchmarks safety

arXiv AI

Jul 29

Rethinking Likelihood distributions: Student's t Likelihood Boosts Bayesian Neural Network Performance

arXiv:2607. 25376v1 Announce Type: cross Abstract: In Bayesian neural networks (BNNs), variational inference is a widely adopted framework for modeling uncertainty in a distributional way, with the evidence lower bound (ELBO) serving as the standard objective function.

By Pei-Hsuan Hsia, Lars H. Heyen, Arvid Weyrauch, Markus Goetz, Achim Streit, Sebastian Krumscheid, Charlotte Debus

arXiv Machine Learning

Jul 7

Toward Efficient Uncertainty in LLMs through Evidential Knowledge Distillation

arXiv:2507. 18366v2 Announce Type: replace Abstract: Accurate uncertainty quantification remains a key challenge for standard LLMs, prompting the adoption of Bayesian and ensemble-based methods.

By Lakshmana Sri Harsha Nemani, P. K. Srijith, Tomasz Ku\'smierczyk

llms fine-tuning efficiency

arXiv AI

Jun 2

What Makes a Strong Model? A Unified Spectral Analysis of Knowledge Transfer over High-dimensional Linear Regression

arXiv:2606. 01292v1 Announce Type: cross Abstract: Teacher-Student Knowledge Transfer (KT) is ubiquitous in modern machine learning, ranging from classical model compression via Knowledge Distillation (KD) to the emergent phenomenon of Weak-to-Strong (W2S) generalization.

By Wendao Wu, Fangqing Zhang, Haihan Zhang, Cong Fang

efficiency

arXiv Machine Learning

Jun 11

Beyond Dark Knowledge: Mixup-Based Distillation for Reliable Predictions

arXiv:2606. 12171v1 Announce Type: cross Abstract: Knowledge Distillation (KD) and mixup have proven effective at inducing smoothness in class boundaries; KD captures inherent class relationships in probability distributions, and mixup enforces them through convex combinations of inputs.

By Jos\'e Medina, Paul Honeine, Abdelaziz Bensrhair, Amnir Hadachi

efficiency

One email a morning, machine-written

Related stories

Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models

Discovering and Preserving Category Correlation Knowledge via Adaptive Reciprocal Knowledge Distillation

Rethinking Likelihood distributions: Student's t Likelihood Boosts Bayesian Neural Network Performance

Toward Efficient Uncertainty in LLMs through Evidential Knowledge Distillation

What Makes a Strong Model? A Unified Spectral Analysis of Knowledge Transfer over High-dimensional Linear Regression

Beyond Dark Knowledge: Mixup-Based Distillation for Reliable Predictions