IntBMoE introduces a block‑conditioned mixture‑of‑experts that decouples participation, execution, and materialization by combining dense expert composition with sparse block execution. Each internal layer uses a lightweight hypernetwork to merge all expert bases into a single composed expert, while a router selects only a few blocks per token, keeping compute and memory costs low. Experiments on image classification, language modeling, and sequential recommendation demonstrate consistent performance gains, and the model is deployed in AMap’s generative recommendation system, improving UVCTR by 2.4% in online A/B tests.
By Ran Cheng, Longfei Xu, Zheng Liu, Kaikui Liu, Xiangxiang Chu
arXiv:2607. 08511v1 Announce Type: new Abstract: Choosing a learning rate scheduling strategy is critical to neural network training, but manual selection is costly and rarely exhaustive.
By Hafsa Mateen, Radu Timofte, Dmitry Ignatov
Choosing a learning rate scheduling strategy is critical to neural network training, but manual selection is costly and rarely exhaustive. While classical AutoML approaches often treat the scheduler as a secondary hyperparameter, we systematically investigate its impact on classification accuracy across a diverse pool of architectures.
arXiv:2503. 05641v4 Announce Type: replace-cross Abstract: Combining existing pre-trained LLMs is a promising approach for diverse reasoning tasks.
By Justin Chih-Yao Chen, Sukwon Yun, Elias Stengel-Eskin, Tianlong Chen, Mohit Bansal
arXiv:2609.36222v1 Announce Type: new
Abstract: Large language models are increasingly expensive to serve. In large-scale serving systems, autoregressive decoding is often bottlenecked by transferrin...
By Ali Abbasi, Justin Shi, Soheil Kolouri
arXiv:2608. 15299v1 Announce Type: cross Abstract: Sparsely-activated Mixture-of-Experts (MoE) Transformers universally fix the same number of routed experts across all layers, a convention that ignores the well-documented heterogeneity in layer-wise redundancy.
By Lie Li, Wen Li, Junxiao Shen, Gusheng Hu
The paper studies how the design of Mixture-of-Experts (MoE) routers affects inference speed when combined with Speculative Decoding (SD). It shows that routers promoting high expert coactivation reduce memory transfer costs and improve runtime. By integrating a global load‑balancing loss, shared experts, a consistency loss, and an autoregressive expert selection mechanism, the authors achieve a 21% throughput gain over baseline MoEs while preserving accuracy.
By Kumari Nishu, Han-Byul Kim, Santosh Chilkunda, Maxwell Horton, Arnav Kundu, Mohammad Samragh, Lauren Hannah, Mohammad Sekhavat, Nikhil Bhendawade, Manuel Ciosici, Iman Mirzadeh, Keivan Alizadeh Vahid, David Harrison, Irina Belousova, Mehrdad Farajtabar, Minsik Cho
arXiv:2609.39350v1 Announce Type: cross
Abstract: As model sizes continue to scale, distributed training has become inevitable. Automatic parallelization techniques can derive efficient training para...
By Mengyuan Fan, Peizhuang Cong, Zixiao Huang, Si Xu, Tong Qiao, Yanghao Li, Jing Yang, Tong Yang, Quanlu Zhang, Yu Wang
arXiv:2607. 15745v1 Announce Type: new Abstract: Common practice when training Convolutional Neural Networks (CNNs) is to use randomly shuffled mini-batches.
By Anxhelo Shehu, Enes Stastoli, Arben Cela
The paper introduces a MeanField surrogate model for predicting performance of concurrent heterogeneous AI inference workloads on shared GPUs, reducing profiling complexity from combinatorial to linear in the number of models. Experiments with up to six models show high accuracy (R²≈0.96) and efficient integration into a genetic algorithm scheduler, achieving near-exhaustive search performance with minimal runtime overhead.
By Youssef Ennouri, Soonhoi Ha
arXiv:2606. 01007v1 Announce Type: cross Abstract: Sparsely activated Mixture-of-Experts (MoE) models scale capacity via conditional computation, but distributed inference suffers from cross-GPU expert communication and routing-induced load imbalance.
By Zhiyao Xu, Aoxue Liu, Zhanjie Ding, Dan Zhao, Yong Jiang, Qing Li
arXiv:2608. 15383v1 Announce Type: new Abstract: Sparse mixture-of-experts (MoE) language models reduce arithmetic by activating only a small subset of experts per token, yet deployment still requires storing and moving the full expert bank.
By Amjad Saab