arXiv Machine Learning
DAOP: Data-Aware Offloading and Predictive Pre-Calculation for Efficient MoE Inference
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Sep 30
Mira: Memory-Efficient MoE Inference Using Adaptive Caching and Predictive Expert Staging
arXiv:2609.38090v1 Announce Type: new Abstract: Mixture-of-Experts (MoE) models are a compelling architecture for scaling model capacity, making them especially attractive for deployment on resource-...
arXiv AI
Oct 19
Efficient CPU-GPU collaborative MoE inference via Adaptive Expert Assignment
arXiv:2610. 23006v1 Announce Type: new Abstract: Mixture-of-Experts (MoE) models reduce inference computation through sparse expert activation, but their large expert memory footprint often exceeds GPU capacity, making expert offloading necessary for memory-constrained inference.
arXiv AI
Jul 29
SpecPrefetch: Parameter-Efficient Expert Prefetching for Sparse MoE Foundation Models
arXiv:2607. 24787v1 Announce Type: new Abstract: Sparse Mixture-of-Experts (MoE) models expand foundation model capacity through conditional expert activation, but their full expert pools remain difficult to deploy under limited accelerator memory.
arXiv AI
Aug 13
APEX: Adaptive Expert Prefetching for Memory-Efficient Edge MoE Inference
arXiv:2608. 11688v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) models are attractive for edge deployment because they provide high model capacity while activating only a small subset of parameters per token, improving compute efficiency.
arXiv Machine Learning
Oct 21
Specter: Fast MoE Inference via Quantization-Enabled Multi-Step Speculative Offloading
Specter is an inference system for mixture-of-experts (MoE) models that uses quantization to enable speculative decoding and expert offloading. By running a lightweight quantized draft model on the device, Specter predicts future expert usage and pre‑organizes expert movement, reducing PCIe I/O bottlenecks. The approach yields an average 1.50× speedup (up to 1.88×) in time per output token compared to state‑of‑the‑art systems, making single‑GPU MoE inference more practical on commodity hardware.
arXiv Machine Learning
Jun 2
ViBE: Co-Optimizing Workload Skew and Hardware Variability for MoE Serving
arXiv:2606. 00735v1 Announce Type: cross Abstract: In distributed Mixture-of-Experts (MoE) inference, input-dependent token routing interacts with GPU performance variability to create persistent stragglers under synchronized execution, where the slowest GPU determines layer latency.