arXiv Machine Learning July 15, 2026 By Zijie Liu, Jie Peng, Jinhao Duan, Zirui Liu, Kaixiong Zhou, Mingfu Liang, Luke Simon, Xi Liu, Zhaozhuo Xu, Tianlong Chen

A Replicate-and-Quantize Strategy for Plug-and-Play Load Balancing of Sparse Mixture-of-Experts LLMs

Read the original on arXiv Machine Learning →

arXiv:2602. 19938v2 Announce Type: replace Abstract: Sparse Mixture-of-Experts (SMoE) architectures are increasingly used to scale large language models efficiently, delivering strong accuracy under fixed compute budgets.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv AI

Aug 17

FreeBalance: Pre-Routing Online Moe Load Balancing via Residual Workload Prediction

arXiv:2608. 14205v1 Announce Type: new Abstract: Load imbalance poses a major bottleneck to the efficiency of expert parallelism in distributed inference of Mixture-of-Experts (MoE) models.

By Pengfei Chen, Yize Wu, Shouxu Kuang, Ke Gao, Ling Li

arXiv Computation and Language

Sep 10

Preserving Long-Tailed Expert Information in Mixture-of-Experts Tuning

arXiv:2604.23036v2 Announce Type: replace-cross Abstract: Despite MoE models leading many benchmarks, supervised fine-tuning (SFT) for the MoE architectures remains difficult because its router layer...

By Haoze He, Xingyuan Ding, Xuan Jiang, Xinkai Zou, Alex Cheng, Yibo Zhao, Juncheng Billy Li, Heather Miller

fine-tuning benchmarks safety

arXiv Machine Learning

Aug 27

Ban&Pick: Enhancing Performance and Efficiency of MoE-LLMs via Smarter Routing

The paper introduces Ban&Pick, a post‑training, plug‑and‑play routing strategy for Sparse Mixture‑of‑Experts large language models. It identifies and reinforces a small group of highly influential experts while dynamically pruning redundant ones, leading to accuracy gains across math, code, and reasoning benchmarks. Experiments on DeepSeek and Qwen3 show notable performance improvements and a 1.25× inference speedup without retraining or architectural changes.

By Yuanteng Chen, Peisong Wang, Yuantian Shao, Nanxin Zeng, Chang Xu, Jian Cheng

llms benchmarks

arXiv Computation and Language

Sep 10

Distribution-Consistent Inference for Dynamic Sparse Mixture-of-Experts

arXiv:2609.09241v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) architectures have emerged as a powerful paradigm for scaling model capacity while preserving efficient inference in large f...

By Dohyeon Kim, Bedionita Soro, Sung Ju Hwang

llms benchmarks safety

arXiv AI

Jul 28

LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in Large Language Models

arXiv:2411. 00918v5 Announce Type: replace-cross Abstract: Mixture of experts (MoE) architectures have become a cornerstone for scaling up and are a key component in most large language models such as GPT-OSS, DeepSeek-V3, Llama-4, and Gemini-2.

By Nam V. Nguyen, Thong T. Doan, Luong Tran, Van Nguyen, Quang Pham

llms benchmarks

arXiv Machine Learning

4d ago

BASE: Batch-Aware Selection of Experts Using Predicted Removal Error for Efficient MoE Decoding

arXiv:2609.36222v1 Announce Type: new Abstract: Large language models are increasingly expensive to serve. In large-scale serving systems, autoregressive decoding is often bottlenecked by transferrin...

By Ali Abbasi, Justin Shi, Soheil Kolouri

llms efficiency