Instruction-tuned models are deployed into environments where domains are heterogeneous and evolve, yet adding new domains or data typically requires costly retraining. We present AutoAdapt, a modular...
AutoAdapt is a modular framework that automatically discovers latent domains and trains per-domain Low‑Rank Adaptation (LoRA) adapters independently in parallel, without modifying other adapters. It uses parameter‑free routing to integrate new domains, achieving performance comparable to a LoRA adapter trained on all domains while avoiding full‑model retraining. The approach prevents domain interference by construction, enabling taxonomy‑free domain specialization across 14 benchmarks and GPT‑4o evaluations.
By Josh McGiff, Salma Mekaoui, Robert Shanahan, Nikola S. Nikolov
The paper investigates whether the sparsity of Mixture-of-Experts (MoE) models leads to intrinsic semantic organization across modalities and domains. It shows that experts naturally specialize semantically even without explicit modular training. The authors propose ExpertLens, a data‑free method that decodes router weights to identify domain‑specialized experts, enabling selective fine‑tuning that matches or exceeds full fine‑tuning while updating only 21.7–47.0% of parameters and achieving a 4.0× speedup, outperforming LoRA in both performance and efficiency.
By Damiano Marsili, Raphi Kang, Aditya Mehta, Pietro Perona, Georgia Gkioxari
arXiv:2606. 06098v1 Announce Type: cross Abstract: Foundational Large Language Models (LLMs) demonstrate proficiency on a wide range of general tasks, and achieve remarkable results on various specialized tasks via domain-expert LLMs.
By Eros Fan\`i, O\u{g}uzhan Ersoy
arXiv:2606. 31413v1 Announce Type: new Abstract: Composing independently trained LoRA adapters into a single large language model is useful for multi-domain adaptation, especially when the original training data cannot be shared.
By Seyed Alireza Molavi, Zhan Su, Yan Hu, Peyman Sheikholharam Mashhadi, Stefan Byttner, Prayag Tiwari
arXiv:2610.34756v1 Announce Type: cross
Abstract: Low-Rank Adaptation (LoRA) has emerged as the dominant paradigm for parameter-efficient fine-tuning of large language models. Standard all-layer LoRA...
By Fei Zuo, Yizhou Huang, Kezhi Wang