arXiv Machine Learning

Decoupled and Distilled: Task-Adaptive LoRA-Teachers with Ensemble Knowledge Transfer for Few-Shot Class-Incremental Learning

The paper introduces TALON, a Task‑Adaptive LoRA‑Teacher framework for Few‑Shot Class‑Incremental Learning. TALON assigns a dedicated LoRA‑Teacher to each incremental task, then distills the frozen teachers into a single LoRA‑Student via Ensemble Knowledge Transfer, using a semantic‑guided weighting scheme to reduce forgetting and overfitting. Experiments on four FSCIL benchmarks show that TALON matches or surpasses state‑of‑the‑art accuracy while using up to 33× fewer deployment parameters and cutting inference time by 41.7%.

arXiv Machine Learning

2d ago

Dynamic LoRA-Experts and Prototype-Ensemble Matching for Class-Incremental Learning

arXiv:2609.39839v1 Announce Type: new Abstract: Class-Incremental Learning (CIL) aims to continuously learn new classes without forgetting previously acquired knowledge. Parameter-efficient fine-tuni...

By Hongwei Zhao (School of Computer Science,Engineering, Beihang University), Rui Liu (School of Computer Science,Engineering, Beihang University), Yansong Liu (School of Computer Science,Engineering, Beihang University)

fine-tuning benchmarks safety

arXiv Computation and Language

Sep 10

On-Policy Distillation for Vision-Language Model Adaptation, an Effective Paradigm on Low-Quality Multimodal Data

arXiv:2609.10321v1 Announce Type: new Abstract: Knowledge distillation offers an efficient route to transfer a task-adapted vision-language teacher to a compact student. The training target in curren...

By Hongyuan Zhang, Xianda Guo, Yanlun Peng, Qianlong Yang, Yubin Guo, Pinhan Fu, Mulin Chen, Xiaozhen Qiao, Ping Luo

llms efficiency multimodal benchmarks

arXiv Machine Learning

Sep 15

Discovering and Preserving Category Correlation Knowledge via Adaptive Reciprocal Knowledge Distillation

arXiv:2609.13199v1 Announce Type: new Abstract: Knowledge distillation aims to improve the performance of lightweight student models by transferring knowledge from larger and more powerful teacher mo...

By Dawen Jiang, Zhishu Shen, Zeyu Liu, Tiehua Zhang

efficiency benchmarks safety

arXiv Machine Learning

Sep 1

One Adapter, Many Tasks: Task-Conditioned Feature Transformations for Continual Learning

arXiv:2608.31096v1 Announce Type: cross Abstract: Class-incremental learning (CIL) requires a model to incrementally learn tasks that contain new classes without accessing earlier training data while...

By Yunxiang Fu, Meng Lou, Yizhou Yu

fine-tuning

arXiv AI

Jul 28

Continual Knowledge Consolidation LORA for Domain Incremental Learning

arXiv:2510. 16077v2 Announce Type: replace-cross Abstract: Domain Incremental Learning (DIL) is a sub-branch of continual learning that aims to address the never-ending arrival of new domains without catastrophic forgetting.

By Naeem Paeedeh, Mahardhika Pratama, Weiping Ding, Jimmy Cao, Wolfgang Mayer, Ryszard Kowalczyk, Ary Shiddiqi

fine-tuning benchmarks safety

arXiv AI

Jun 19

Wisdom of Committee: Diverse Distillation from Large Foundation Models and Domain Experts

arXiv:2402. 14035v4 Announce Type: replace-cross Abstract: Knowledge distillation from foundation models to compact domain models is challenging due to substantial gaps in capacity, architecture, and modality.

By Zichang Liu, Qingyun Liu, Yuening Li, Liang Liu, Anshumali Shrivastava, Shuchao Bi, Lichan Hong, Ed H. Chi, Zhe Zhao

llms efficiency

arXiv Machine Learning

Aug 12

GLAM: Efficient Continual Learning at Scale via Grouped LoRA Adapter Merging

arXiv:2509. 13211v4 Announce Type: replace Abstract: The ability to learn continuously over time remains a major challenge for modern machine learning systems, even in the era of Foundation Models.

By Irene Testa, Luigi Quarantiello, Eric Nuertey Coleman, Samrat Mukherjee, Julio Hurtado, Vincenzo Lomonaco

fine-tuning benchmarks

arXiv AI

Jun 30

Fisher-Routed Mixture of Experts for Federated Class-Incremental Learning

arXiv:2606. 28835v1 Announce Type: cross Abstract: Federated Learning (FL) emerged as a promising distributed machine learning paradigm.

By Wenhao Yuan, Chenchen Lin, Jian Chen, Jinfeng Xu, Zewei Liu, Edith Cheuk Han Ngai

benchmarks

arXiv Machine Learning

Aug 4

Tevatron Meets Megatron: Expert-Parallel LLM Reranker Training on an Academic Budget

arXiv:2608. 00916v1 Announce Type: cross Abstract: Modern reranking recipes---billion-scale cross-encoders, mixture-of-experts (MoE) backbones, and distillation against strong teachers---have outpaced the training infrastructure available to most academic groups.

By Zhichao Xu, Xueguang Ma, Shengyao Zhuang, Luyu Gao, Wenqian Ye, Yu Wang, Jamie Callan, Jimmy Lin

llms rag fine-tuning efficiency benchmarks data-engineering

arXiv AI

3d ago

Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation

arXiv:2605.07111v3 Announce Type: replace-cross Abstract: Recent literature on fine-tuning Large Language Models highlights a fundamental debate. While Full Fine-Tuning (FFT) provides greater represe...

By Haozhan Tang, Xiuqi Zhu, Xinyin Zhang, Boxun Li, Virginia Smith, Kevin Kuo

llms fine-tuning data-engineering

arXiv AI

Jun 18

From Memorization to Parameter Interference: How Overtraining Experts Harms Model Merging

arXiv:2506. 14126v2 Announce Type: replace-cross Abstract: Modern deep learning is increasingly characterized by the use of open-weight foundation models that can be fine-tuned on specialized datasets.

By Stefan Horoi, Guy Wolf, Eugene Belilovsky, Gintare Karolina Dziugaite

fine-tuning

arXiv AI

Jul 1

Knowledge Distillation from Large Reasoning Models to Compact Student Models: A Case Study on the John O Bryan Mathematics Competition

arXiv:2606. 31048v1 Announce Type: cross Abstract: This paper investigates knowledge distillation from a large reasoning model (DeepSeek-R1) to a compact student model (Qwen2.

By Gaurab Baral, Aaditya Khanal, Yangyang Tao, Junxiu Zhou

agents fine-tuning efficiency benchmarks

Related stories