arXiv Machine Learning
FluxMoE: Decoupling Expert Residency for High-Performance MoE Serving
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Aug 18
Every Expert Counts: ExactMoE for Memory-Efficient W4A16 Inference
arXiv:2608. 15383v1 Announce Type: new Abstract: Sparse mixture-of-experts (MoE) language models reduce arithmetic by activating only a small subset of experts per token, yet deployment still requires storing and moving the full expert bank.
arXiv AI
Jul 28
DraftExpert: Expansion-Aware Self-Speculative Decoding for End-Device MoE Inference
arXiv:2607. 24434v1 Announce Type: cross Abstract: Large Mixture-of-Experts (MoE) language models are attractive for end-device deployment because only a small subset of experts is active per token, but their routed expert weights often exceed accelerator memory.
arXiv Machine Learning
Aug 31
Characterization of Request and Token Energy Costs for LLM Inference Workloads on GPU Platforms
arXiv:2608.28044v1 Announce Type: cross Abstract: Large language model (LLM) inference serving is priced by tokens, but GPU energy is consumed over inference windows. This accounting mismatch makes t...
arXiv Machine Learning
Aug 27
ExFold: Unified Expert Folding for Training-Free MoE Prefill-Decode Acceleration
ExFold is a training‑free expert‑folding framework that jointly accelerates the prefill and decode phases of Mixture‑of‑Experts (MoE) models by projecting the contributions of excluded experts onto a retained expert set using calibrated scalar projectors. It treats both phases as a budgeted output‑approximation problem, achieving token‑level Top‑K folding for prefill and batch‑level expert‑pool folding for decode. Implemented as a plug‑and‑play plugin in vLLM with a lightweight CUDA kernel, ExFold delivers up to 1.41× TTFT and 2.45× TPOT speedups while preserving about 99% of the original model quality.
Hugging Face Trending Papers
Jul 27
DraftExpert: Expansion-Aware Self-Speculative Decoding for End-Device MoE Inference
Large Mixture-of-Experts (MoE) language models are attractive for end-device deployment because only a small subset of experts is active per token, but their routed expert weights often exceed accelerator memory. We target latency-critical single-user settings where routed experts are staged on demand from CPU memory to a GPU or from Flash to a mobile NPU.
arXiv Machine Learning
Sep 1
WiSP: A Working-Set View of Mixture-of-Experts Serving on Extremely Low-Resource Hardware
WiSP (Working‑Set Paging) is a routing‑aware expert pager that allows Mixture‑of‑Experts models to run on GPUs that cannot hold the entire expert pool by paging experts in and out of VRAM while preserving byte‑identical outputs. On a 24 GiB RTX 3090, WiSP doubles decode throughput compared to static offload when the model does not fit, and its companion policy MV‑WSA allocates VRAM between resident experts and KV cache based on marginal latency benefit, reducing end‑to‑end time by up to 1.19× without altering model outputs.