arXiv Computer Vision October 2, 2026 By Damiano Marsili, Raphi Kang, Aditya Mehta, Pietro Perona, Georgia Gkioxari

Harnessing Domain Specialists in Multimodal Mixture-of-Experts for Efficient Adaptation

Read the original on arXiv Computer Vision →

The paper investigates whether the sparsity of Mixture-of-Experts (MoE) models leads to intrinsic semantic organization across modalities and domains. It shows that experts naturally specialize semantically even without explicit modular training. The authors propose ExpertLens, a data‑free method that decodes router weights to identify domain‑specialized experts, enabling selective fine‑tuning that matches or exceeds full fine‑tuning while updating only 21.7–47.0% of parameters and achieving a 4.0× speedup, outperforming LoRA in both performance and efficiency.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computer Vision.

arXiv AI

Jun 8

MACS: Modality-Aware Capacity Scaling for Efficient Multimodal MoE Inference

arXiv:2605. 05225v3 Announce Type: replace-cross Abstract: Mixture-of-Experts Multimodal Large Language Models (MoE MLLMs) suffer from a significant efficiency bottleneck during Expert Parallelism (EP) inference due to the straggler effect.

By Bo Li, Chuan Wu, Shaolin Zhu

llms multimodal benchmarks

arXiv Machine Learning

Sep 23

From Experts to Sub-experts: Fine-grained Parameter-Efficient Fine-Tuning for MoE LLMs

arXiv:2609.25655v1 Announce Type: new Abstract: As large language models (LLMs) scale rapidly, dense full-parameter adaptation becomes increasingly expensive, motivating sparse and modular architectu...

By Zhentao Tan, Chang Liu, Yao Liu, Yue Wu, Jieping Ye

llms fine-tuning benchmarks

arXiv Machine Learning

Jun 30

Symbiotic-MoE: Unlocking the Synergy between Generation and Understanding

arXiv:2604. 07753v2 Announce Type: replace-cross Abstract: Empowering Large Multimodal Models (LMMs) with image generation often leads to catastrophic forgetting in understanding tasks due to severe gradient conflicts.

By Xiangyue Liu, Zijian Zhang, Miles Yang, Zhao Zhong, Liefeng Bo, Ping Tan

llms diffusion multimodal benchmarks

Hugging Face Trending Papers

Sep 22

From Experts to Sub-experts: Fine-grained Parameter-Efficient Fine-Tuning for MoE LLMs

As large language models (LLMs) scale rapidly, dense full-parameter adaptation becomes increasingly expensive, motivating sparse and modular architectures such as Mixture-of-Experts (MoE) models. This...

llms fine-tuning benchmarks

arXiv Machine Learning

Aug 18

Self-Routed Tensor Adapters for Parameter-Efficient Universal Visual Adaptation

arXiv:2608. 16384v1 Announce Type: cross Abstract: Universal visual representations require adaptation mechanisms that adapt across heterogeneous domains without fragmenting knowledge into domain-specific modules.

By Suraj Yadav

fine-tuning benchmarks

arXiv Computation and Language

Sep 10

Preserving Long-Tailed Expert Information in Mixture-of-Experts Tuning

arXiv:2604.23036v2 Announce Type: replace-cross Abstract: Despite MoE models leading many benchmarks, supervised fine-tuning (SFT) for the MoE architectures remains difficult because its router layer...

By Haoze He, Xingyuan Ding, Xuan Jiang, Xinkai Zou, Alex Cheng, Yibo Zhao, Juncheng Billy Li, Heather Miller

fine-tuning benchmarks safety

One email a morning, machine-written

Related stories

MACS: Modality-Aware Capacity Scaling for Efficient Multimodal MoE Inference

From Experts to Sub-experts: Fine-grained Parameter-Efficient Fine-Tuning for MoE LLMs

Symbiotic-MoE: Unlocking the Synergy between Generation and Understanding

From Experts to Sub-experts: Fine-grained Parameter-Efficient Fine-Tuning for MoE LLMs

Self-Routed Tensor Adapters for Parameter-Efficient Universal Visual Adaptation

Preserving Long-Tailed Expert Information in Mixture-of-Experts Tuning