← Back to all news
arXiv Machine Learning

How Far Can Foundation Models Be Distilled for Efficient Jet Tagging?

The paper investigates how well foundation models can be distilled for efficient jet tagging at the Large Hadron Collider. Six teacher models, varying in architecture, symmetry, and pretraining—including two foundation models—are distilled into five small student models. The strongest distilled students, Transformer and L‑GATr‑slim, achieve state‑of‑the‑art performance with only 20,000 parameters, and even a 2,000‑parameter version matches a much larger model, ParT, while offering significantly lower inference cost.

Related stories

arXiv Computation and Language
Oct 28

Strong Teacher Not Needed? On Distillation in LLM Pretraining

The paper investigates whether a strong teacher is necessary for knowledge distillation in large language model pretraining. By varying teacher and student sizes and token budgets, the authors create strong‑to‑weak, same‑level, and weak‑to‑strong teacher‑student pairs and evaluate distillation effectiveness. They find that even small, undertrained teachers can improve larger students when losses are properly mixed, and that pushing teachers to be stronger can saturate or reverse gains, while distillation mainly boosts generalization rather than in‑domain fitting.

By Taiming Lu, Zhuang Liu
llmsefficiency
More like this →
arXiv Machine Learning
Oct 3

Distillation of Tabular Foundation Models into Efficient Predictors

The paper presents a method for distilling tabular foundation models (TFMs) into lightweight, dataset‑specific students. By using the full labeled training set as teacher context and training students on both observed and synthetic queries, the authors achieve significant performance gains over traditional supervised models on TabArena and TALENT benchmarks. The distilled students also provide substantial inference speedups, reducing the cost of repeated inference.

By Minho Jeong, Dooho Lee, Jinmo Lee, Jaemin Yoo
llmsefficiency
More like this →
arXiv Machine Learning
Oct 9

KDFP: A first-principles approach to knowledge distillation in large language models

KDFP is a new white‑box methodology for general knowledge distillation in large language models, addressing a gap left by prior work that focused mainly on post‑training abilities such as instruction following and chain‑of‑thought reasoning. The authors evaluate lessons from earlier studies and conduct new experiments to design a distillation approach tailored to modern LLMs. KDFP achieves 1.6%–4.9% higher performance across nine benchmarks and improves training efficiency by up to 99.1% through reduced temporary parameters.

By Ryan Swift, Konstantinos Psounis
llmsefficiencybenchmarks
More like this →
arXiv Machine Learning
Oct 2

Decoupled and Distilled: Task-Adaptive LoRA-Teachers with Ensemble Knowledge Transfer for Few-Shot Class-Incremental Learning

The paper introduces TALON, a Task‑Adaptive LoRA‑Teacher framework for Few‑Shot Class‑Incremental Learning. TALON assigns a dedicated LoRA‑Teacher to each incremental task, then distills the frozen teachers into a single LoRA‑Student via Ensemble Knowledge Transfer, using a semantic‑guided weighting scheme to reduce forgetting and overfitting. Experiments on four FSCIL benchmarks show that TALON matches or surpasses state‑of‑the‑art accuracy while using up to 33× fewer deployment parameters and cutting inference time by 41.7%.

By Hongwei Zhao (School of Computer Science,Engineering, Beihang University), Rui Liu (School of Computer Science,Engineering, Beihang University), Yansong Liu (School of Computer Science,Engineering, Beihang University), Zhiyuan Zou (School of Computer Science,Engineering, Beihang University), Yong Chen (School of Computer Science, Beijing University of Posts,Telecommunications)
fine-tuningefficiencybenchmarks
More like this →