← Back to all news
arXiv AI By Liangwei Nathan Zheng, Wenhao Liang, Wei Emma Zhang, Lin Yue, Weitong Chen

Tackling Multimodal Learning Challenges with Mixture-of-Expert: A Survey

Read the original on arXiv AI →

The survey titled "Tackling Multimodal Learning Challenges with Mixture-of-Expert: A Survey" reviews how Mixture-of-Experts (MoE) frameworks address multimodal learning problems. It categorizes MoE’s contributions into three roles: an efficient multimodal engine that scales without excessive computation, a representation learner that fuses diverse expert knowledge, and an adapter that handles imperfect data such as missing modalities. The paper also highlights research gaps like interpretable routing, expert communication, and lifelong multimodal learning.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Machine Learning
Sep 22

Generalized Multimodal Foundation Model

The paper introduces a generalized multimodal foundation model that can handle arbitrary combinations of modalities and prediction tasks. It trains on large-scale synthetic multimodal datasets with diverse causal structures to learn transferable multimodal correlations. Experiments on 18 real-world datasets across 12 modalities and 11 tasks show competitive performance compared to specialized models without task-specific adaptation.

By Huizi Cui, Zongbo Han, Chenggong Ding, Naichuan Xiao, Jialong Yang, Jingdong Chen, Guangyu Wang, Qinghua Hu, Changqing Zhang
multimodal
More like this →
arXiv Computer Vision
Oct 19

Mitigating Task Objective Conflicts for Continual Learning in Unified Multimodal Models

The paper introduces UniDecouple, a selective decoupling framework for continual learning in unified multimodal models that addresses Task Objective Conflict (TOC). By employing a Task-Aware Mixture-of-Experts (TA‑MoE) architecture, it retains shared attention layers while adding task‑specific feed‑forward network experts to reduce optimization conflicts. Hierarchical expert routing and hybrid collaboration further balance task specialization with cross‑task knowledge sharing, and a two‑stage training strategy enhances both multimodal understanding and image generation performance.

By Jiaxing Zhang, Hao Tang
diffusionmultimodalbenchmarks
More like this →