The paper demonstrates that Mixture-of-Experts (MoE) models can avoid the need for calibration data when selecting which expert channels to load during decoding. Instead of using a calibrated threshold, a top‑k selection strategy—choosing the k highest‑scoring channels per token—achieves comparable or better performance without calibration. Additionally, the authors propose a reader that uses a ternary sketch and low‑rank mapping to recover skipped channels, achieving 98.1% of full‑model accuracy while reading only 17.8% of expert weights and speeding up inference by up to 2.69× compared to optimized vLLM.
By Geunhyoung Kim, Sung Ju Hwang
arXiv:2609.36222v1 Announce Type: new
Abstract: Large language models are increasingly expensive to serve. In large-scale serving systems, autoregressive decoding is often bottlenecked by transferrin...
By Ali Abbasi, Justin Shi, Soheil Kolouri
arXiv:2610.28369v1 Announce Type: cross
Abstract: A growing body of work proposes dynamic expert-skipping methods for Mixture-of-Experts (MoE) models that adjust the number of experts activated per t...
By Jakob Steimle, Thomas Elsken, Daniel Ehrhardt, Lukas Rinder, Daniel Cremers, Xi Wang
Prefill‑only decision models evaluate every candidate in a menu in a single forward pass, avoiding decoding and reducing cost by one to two orders of magnitude compared to generative language models. The paper demonstrates that when only the candidate menu changes, the model’s post‑intervention accuracy can be predicted solely from the cached first‑pass distribution using a simple estimator that renormalizes and selects the argmax, without any labels or second pass. Across seven model families, ten datasets, and two task types, this menu‑only intervention prediction is within 4.2 points of actual accuracy, and in one family it is exact, whereas a probability‑level variant fails by 21 points, indicating the property resides in ranking rather than calibrated probabilities.
By Ran Li, Lei Chen
The paper presents a post‑training method for dynamically routed mixture‑of‑experts (MoE) models that integrates independently optimized specialists efficiently. It introduces calibrated multi‑teacher on‑policy distillation (C‑MOPD) with teacher‑wise loss normalization and clipping, combined with a post‑capacity bounded‑width dispatch, final‑route replay, and asynchronous rollout–training overlap. On the Turing‑20B‑A2B model, the approach achieves a high integration score (0.9988), retains most specialist gains, outperforms shared SFT and Qwen3‑8B on reasoning‑enabled benchmarks, and reduces consolidation time by 2.86×.
By Yue He, Jiahui Hu, Shaman Tang, Hua Zhou, Qiman Wu, Menglin Li, Yizhao Wang, Yuheng Zhang, Da Zhu, Zhou Wang, Anhua Liu, Hanlin Chen, Mingyuan Wang, Rui Xin, Yuhua Wei, Shuang Su, Siqi Liu, Kai Yang, Weichao Huang, Hang Zhang, Honggou Yang, Xianming Liu
arXiv:2608. 07911v1 Announce Type: new Abstract: Mixture-of-Experts (MoE) models have outgrown accelerator memory, and offloading expert weights to host memory is now standard.
By Yu Zhang