arXiv Machine Learning August 28, 2026 By Sameh Khattab, Jean-Philippe Corbeil, Osman Alperen \c{C}inar-Kora\c{s}, Amin Dada, Julian Friedrich, Jiawei He, Douglas Teodoro, Jens Kleesiek

Modular Expert Merging for Biomedical Retrieval

Read the original on arXiv Machine Learning →

The paper proposes a method called Modular Expert Merging for Biomedical Retrieval, which combines independently trained domain‑specialized experts instead of large mixed‑domain training. Experiments across four decoder‑only LLM families (0.6B‑7B) and twelve retrieval tasks from MTEB show that merging experts consistently outperforms mixed‑domain training. The authors also introduce a Synthesize‑Train‑Merge (STM) framework that generates hard negatives with a top‑tier LLM, fine‑tunes experts via LoRA, and merges them, achieving strong biomedical retrieval performance while retaining competitive general‑domain results.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv AI

Aug 25

MixLoRA-DSI: Dynamically Expandable Mixture-of-LoRA Experts for Rehearsal-Free Generative Retrieval over Dynamic Corpora

arXiv:2507.09924v2 Announce Type: replace-cross Abstract: Continually updating model-based indexes in generative retrieval with new documents remains challenging, as full retraining is computationall...

By Tuan-Luc Huynh, Thuy-Trang Vu, Weiqing Wang, Trung Le, Dragan Ga\v{s}evi\'c, Yuan-Fang Li, Thanh-Toan Do

fine-tuning

arXiv AI

Sep 17

Generalist-Specialist Mixture-of-Experts for Rare Pathology Detection in Multimodal Imaging

The paper introduces Generalist‑Specialist Mixture‑of‑Experts (GS‑MoE), a two‑branch architecture that combines a cross‑modal generalist model with modality‑specific specialists through domain‑constrained feature fusion. GS‑MoE improves detection of rare pathologies in multimodal medical imaging, achieving significant per‑class F1 gains and outperforming dense and specialist‑only MoE baselines while using about 53% fewer active parameters at inference. The study demonstrates that balancing cross‑modal shared representations with expert routing can enhance performance on low‑prevalence conditions.

By Johannes Kaiser, Florian Braunmiller, Daniel R\"uckert, Georgios Kaissis

multimodal

arXiv Machine Learning

Aug 3

MoPET: Parameter-Efficient Mixture-of-Experts for Unified Medical Image Classification

arXiv:2607. 29462v1 Announce Type: cross Abstract: Adapting deep learning models to profound clinical heterogeneity typically relies on parameter-efficient fine-tuning (PEFT) to avoid the severe overfitting associated with full end-to-end network updates.

By Sebastian Doerrich, Daniel W\"urtinger, Francesco Di Salvo, Shyam Nandan Rai, Christian Ledig

computer-vision fine-tuning benchmarks

arXiv Computer Vision

2d ago

Harnessing Domain Specialists in Multimodal Mixture-of-Experts for Efficient Adaptation

The paper investigates whether the sparsity of Mixture-of-Experts (MoE) models leads to intrinsic semantic organization across modalities and domains. It shows that experts naturally specialize semantically even without explicit modular training. The authors propose ExpertLens, a data‑free method that decodes router weights to identify domain‑specialized experts, enabling selective fine‑tuning that matches or exceeds full fine‑tuning while updating only 21.7–47.0% of parameters and achieving a 4.0× speedup, outperforming LoRA in both performance and efficiency.

By Damiano Marsili, Raphi Kang, Aditya Mehta, Pietro Perona, Georgia Gkioxari

fine-tuning efficiency multimodal

arXiv Machine Learning

Jun 30

Expert-guided Clinical Text Augmentation via Query-Based Model Collaboration

arXiv:2509. 21530v2 Announce Type: replace Abstract: Data augmentation is a widely used strategy to improve model robustness and generalization by enriching training datasets with synthetic examples.

By Dongkyu Cho, Miao Zhang, Rumi Chunara

llms safety

arXiv Computation and Language

Sep 4

Distilled Rapid Embedding Transfer (DRET): Parameter-Efficient Biomedical Domain Adaptation via Priority-Based Embedding Transfer

The paper introduces Distilled Rapid Embedding Transfer (DRET), a parameter‑efficient method that injects biomedical domain knowledge from large specialized models into a smaller general‑purpose model without retraining on the original specialized corpora. DRET evolves through iterative strategies—tokenizer‑merge (DRET 1.x), hybrid embedding averaging (DRET 2.0), priority‑based embedding transfer (DRET 3.x), and further refinements (DRET 4.x)—and demonstrates that a 66‑million‑parameter DistilBERT can achieve competitive or superior performance on token‑level PICO classification compared to much larger models, while remaining lightweight. The authors validate the embedding‑level transfer with cosine similarity, semantic‑shift, and t‑SNE analyses, highlighting DRET’s potential for scalable, resource‑efficient biomedical text mining.

By Girish Sundaram, Daniel Berleant

llms rag nlp fine-tuning