← Back to all news
arXiv Machine Learning By Ran Cheng, Longfei Xu, Zheng Liu, Kaikui Liu, Xiangxiang Chu

IntBMoE: Integrating Block-Level Conditioning into Expert Composition for Full-Participation Mixture-of-Experts

Read the original on arXiv Machine Learning →

IntBMoE introduces a block‑conditioned mixture‑of‑experts that decouples participation, execution, and materialization by combining dense expert composition with sparse block execution. Each internal layer uses a lightweight hypernetwork to merge all expert bases into a single composed expert, while a router selects only a few blocks per token, keeping compute and memory costs low. Experiments on image classification, language modeling, and sequential recommendation demonstrate consistent performance gains, and the model is deployed in AMap’s generative recommendation system, improving UVCTR by 2.4% in online A/B tests.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Aug 27

Ban&Pick: Enhancing Performance and Efficiency of MoE-LLMs via Smarter Routing

The paper introduces Ban&Pick, a post‑training, plug‑and‑play routing strategy for Sparse Mixture‑of‑Experts large language models. It identifies and reinforces a small group of highly influential experts while dynamically pruning redundant ones, leading to accuracy gains across math, code, and reasoning benchmarks. Experiments on DeepSeek and Qwen3 show notable performance improvements and a 1.25× inference speedup without retraining or architectural changes.

By Yuanteng Chen, Peisong Wang, Yuantian Shao, Nanxin Zeng, Chang Xu, Jian Cheng
llmsbenchmarks
More like this →
arXiv Machine Learning
Aug 27

ExFold: Unified Expert Folding for Training-Free MoE Prefill-Decode Acceleration

ExFold is a training‑free expert‑folding framework that jointly accelerates the prefill and decode phases of Mixture‑of‑Experts (MoE) models by projecting the contributions of excluded experts onto a retained expert set using calibrated scalar projectors. It treats both phases as a budgeted output‑approximation problem, achieving token‑level Top‑K folding for prefill and batch‑level expert‑pool folding for decode. Implemented as a plug‑and‑play plugin in vLLM with a lightweight CUDA kernel, ExFold delivers up to 1.41× TTFT and 2.45× TPOT speedups while preserving about 99% of the original model quality.

By Juntong Wu, Yifei Liu, Junyi Chen, Siqi Fan, Chaoran Feng, Minghao Li, Liujie Zhang, Weihang Chen, Li Yuan
More like this →