← Back to all news
arXiv Computation and Language By Rongfeng Wang, Shichao Weng, Zhiqiang Wang, Xinyu Liu, Yang Yi, Peilong Zhou, Hongwei Tang

Meta-Learning Where to Allocate Experts: Task-Conditioned Layer-Wise Compression for MoEs

Read the original on arXiv Computation and Language →

MetaNet is a support‑set controller that predicts, for each layer of a Mixture‑of‑Experts model, an expert‑retention threshold and a bounded routing bias while keeping the backbone, experts, and router frozen. On DeepSeek‑MoE‑16B‑Chat, MetaNet offers a tunable trade‑off between accuracy and expert activation: a conservative setting activates 3.61 experts on average (40% fewer than a fixed k=6) with comparable MMLU accuracy, whereas an aggressive setting activates only 2.28 experts (62% fewer) with a modest accuracy drop. The MMLU‑trained controller also transfers to C‑Eval, activating 2.90 experts on average (52% fewer than fixed k=6) at 0.386 accuracy.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv Machine Learning
Aug 27

Ban&Pick: Enhancing Performance and Efficiency of MoE-LLMs via Smarter Routing

The paper introduces Ban&Pick, a post‑training, plug‑and‑play routing strategy for Sparse Mixture‑of‑Experts large language models. It identifies and reinforces a small group of highly influential experts while dynamically pruning redundant ones, leading to accuracy gains across math, code, and reasoning benchmarks. Experiments on DeepSeek and Qwen3 show notable performance improvements and a 1.25× inference speedup without retraining or architectural changes.

By Yuanteng Chen, Peisong Wang, Yuantian Shao, Nanxin Zeng, Chang Xu, Jian Cheng
llmsbenchmarks
More like this →
arXiv Machine Learning
Aug 27

SAME: Stabilized Mixture-of-Experts for Multimodal Continual Instruction Tuning

The paper introduces SAME (Stabilized Mixture-of-Experts) to address challenges in Multimodal Continual Instruction Tuning (MCIT) for large language models. SAME mitigates router drift by decomposing routing dynamics into orthogonal subspaces and updating only task-relevant directions, while preventing expert drift through curvature‑aware scaling that uses historical input covariance without rehearsal. The method also employs adaptive expert activation to freeze selected experts during training, reducing redundant computation and cross‑task interference, and demonstrates state‑of‑the‑art performance on a new long‑task‑sequence benchmark.

By Zhen-Hao Xie, Jun-Tao Tang, Yu-Cheng Shi, Han-Jia Ye, De-Chuan Zhan, Da-Wei Zhou
llmscomputer-visionfine-tuningmultimodalbenchmarks
More like this →