← Back to all news
arXiv Machine Learning By Juntong Wu, Yifei Liu, Junyi Chen, Siqi Fan, Chaoran Feng, Minghao Li, Liujie Zhang, Weihang Chen, Li Yuan

ExFold: Unified Expert Folding for Training-Free MoE Prefill-Decode Acceleration

Read the original on arXiv Machine Learning →

ExFold is a training‑free expert‑folding framework that jointly accelerates the prefill and decode phases of Mixture‑of‑Experts (MoE) models by projecting the contributions of excluded experts onto a retained expert set using calibrated scalar projectors. It treats both phases as a budgeted output‑approximation problem, achieving token‑level Top‑K folding for prefill and batch‑level expert‑pool folding for decode. Implemented as a plug‑and‑play plugin in vLLM with a lightweight CUDA kernel, ExFold delivers up to 1.41× TTFT and 2.45× TPOT speedups while preserving about 99% of the original model quality.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Sep 21

IntBMoE: Integrating Block-Level Conditioning into Expert Composition for Full-Participation Mixture-of-Experts

IntBMoE introduces a block‑conditioned mixture‑of‑experts that decouples participation, execution, and materialization by combining dense expert composition with sparse block execution. Each internal layer uses a lightweight hypernetwork to merge all expert bases into a single composed expert, while a router selects only a few blocks per token, keeping compute and memory costs low. Experiments on image classification, language modeling, and sequential recommendation demonstrate consistent performance gains, and the model is deployed in AMap’s generative recommendation system, improving UVCTR by 2.4% in online A/B tests.

By Ran Cheng, Longfei Xu, Zheng Liu, Kaikui Liu, Xiangxiang Chu
computer-vision
More like this →
arXiv Machine Learning
Sep 11

FluxMoE: Decoupling Expert Residency for High-Performance MoE Serving

FluxMoE introduces an expert paging system that decouples Mixture-of-Experts (MoE) model experts from permanent GPU residency, allowing dynamic adaptation to available memory. By combining PagedTensor, a bandwidth‑balanced memory hierarchy, and a budget‑aware residency planner, FluxMoE streams expert weights on demand while keeping computations on GPUs. Experiments on GLM‑4.5 and Mixtral‑8×7B‑Instruct show significant throughput gains and reduced time‑per‑output‑token compared to existing inference engines, without compromising model quality.

By Qingxiu Liu, Yongchao He, Runhan Jiang, Zion Wang, Bohan Zhao, Mi Zhang, Patrick P. C. Lee
llmsefficiency
More like this →