← Back to all news
arXiv Machine Learning By Zheng Lin, Shaoke Fang, Yuxin Zhang, Jinfeng Xu, Zihan Fang, Zhe Chen, Wei Ni, Jun Luo, Symeon Chatzinotas

Redundancy Meets Synergy: Dependency-aware Expert Selection for MoE via Submodular Optimization

Read the original on arXiv Machine Learning →

The paper introduces DS-MoE, a framework that selects a compact subset of experts in Mixture-of-Experts models using difference-of-submodular (DS) optimization. It analyzes the loss degradation via a second-order Taylor expansion to uncover redundancy and synergy among experts, and formulates the selection objective to decouple these effects. A majorization-minimization algorithm with monotonicity guarantees efficiently identifies the optimal subset, and experiments show DS-MoE outperforms state‑of‑the‑art baselines.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Oct 22

GaP-MoE : Explicit Expert Specialization in Mixture-of-Experts via Orthogonal Gradient Accumulation and Projection

GaP-MoE introduces a method for explicit expert specialization in Sparse Mixture-of-Experts models by manipulating expert output gradients instead of adding auxiliary loss terms. The technique selects a leader expert from the top‑K routed experts and projects follower experts’ gradients onto the orthogonal complement of the leader’s recent gradients, reducing directional overlap. Experiments on fine‑tuning BERT for GLUE and pretraining GPT‑2 on various corpora show that GaP‑MoE matches or surpasses standard SMoE performance while remaining a drop‑in module.

By Donghyun Jeong, Yoon-Yeong Kim
llmsfine-tuning
More like this →
arXiv AI
Jun 2

DAG-MoE: From Simple Mixture to Structural Aggregation in Mixture-of-Experts

arXiv:2606. 01062v1 Announce Type: new Abstract: Mixture-of-Experts (MoE) models have become a leading approach for decoupling parameter count from computational cost in large language models, yet effectively scaling MoE performance remains a challenge.

By Jiarui Feng, Hanqing Zeng, Karish Grover, Ruizhong Qiu, Yinglong Xia, Qiang Zhang, Qifan Wang, Ren Chen, Dongqi Fu, Jiayi Liu, Zhoukai Zhao, Xiangjun Fan, Benyu Zhang, Yixin Chen
llmsfine-tuning
More like this →
arXiv Machine Learning
Sep 4

Towards a Statistical Understanding of Mixture-of-Experts

The paper presents a statistical framework for Mixture-of-Experts (MoE) models, treating them as localized aggregation systems. It derives oracle risk bounds that separate approximation, expert‑learning, and router‑estimation errors for both dense and sparse routing with evolving experts. The authors also analyze how sparse Top‑K routing balances computational cost with performance, interpret gating geometrically, and explain how shared experts can capture common predictive structure while allowing routed experts to focus on local residuals.

By Siyuan He, Bokai Yang, Jie Hu, Ziwen Gao, Yuhong Yang
More like this →