arXiv Machine Learning October 8, 2026 By Ruwen Fan (Jimmy), Yuezhi Zu (Jimmy), Junru Li (Jimmy), Qingda Hu (Jimmy), Xinjun (Jimmy), Yang, Jiwu Shu, Youyou Lu

Democratizing MoE inference on commodity GPUs with CoMoE

Read the original on arXiv Machine Learning →

The paper introduces CoMoE, a communication‑efficient Mixture‑of‑Experts inference system designed for commodity GPUs. By using host‑centric routing, host‑backed token multicast, and fine‑grained token‑level aggregation, CoMoE reduces communication volume and eliminates global synchronization stalls. Experiments on RTX 5090 GPUs show up to 1.46× throughput improvement, approaching NVLink‑capable A800 performance while costing only 23.4% of the hardware.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv AI

Jul 23

Fine-grained Computation-Communication Overlap via Tile-level Signaling and Scheduling for Mixture-of-Experts

arXiv:2607. 19539v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) architectures increase model capacity without proportionally increasing computation cost and have become a key building block for scaling large language models (LLMs) to trillion-parameter regimes.

By Minyu Cui, Anna Wingkvist, Morgan Ericsson

llms benchmarks

arXiv Machine Learning

Oct 2

Efficient Expert-Parallel Communication on PCIe-Connected Consumer GPUs

ThunderEP is a new communication design for expert parallelism on PCIe-connected consumer GPUs that eliminates relay hops, uses DMA engines to avoid GPU compute contention, and reduces CPU polling overhead. Integrated into vLLM, it outperforms NCCL on RTX 4090 and RTX 5090 GPUs, delivering average speedups of 2.00× for dispatch, 1.53× for combine, and up to 1.66× end‑to‑end over existing MoE inference frameworks.

By Jaehwan Lee, Sangmin Lee, Chaewon Kim, Junsik Shin, Jaejin Lee

llms benchmarks

arXiv AI

Aug 3

Topology-Aware Data Movement for Disaggregated GPU Inference

arXiv:2607. 28633v1 Announce Type: cross Abstract: Disaggregated LLM inference creates a datacenter networking problem that no existing system solves correctly.

By Sanjeev Rao Ganjihal

llms efficiency

arXiv Machine Learning

Sep 30

Mira: Memory-Efficient MoE Inference Using Adaptive Caching and Predictive Expert Staging

arXiv:2609.38090v1 Announce Type: new Abstract: Mixture-of-Experts (MoE) models are a compelling architecture for scaling model capacity, making them especially attractive for deployment on resource-...

By Sanjali Yadav, Bahar Asgari

efficiency benchmarks

arXiv AI

Jun 10

Achieving Cloud-Grade SLOs for Local Mixture-of-Experts Inference through CPU-GPU Hybrid Design

arXiv:2606. 10493v1 Announce Type: cross Abstract: Local deployment of large Mixture-of-Experts (MoE) models falls short of the service quality achieved in cloud-scale environments, even under low-concurrency workloads.

By Wenxin Wang, Yule Hou, Yu Ji, Peng Qu, Youhui Zhang

efficiency

arXiv Computation and Language

Sep 23

TopoCompress: Topology Aware Token Compression Algorithm for Distributed Edge MoE Inference

The paper introduces TopoCompress, a token compression framework designed for distributed edge Mixture-of-Experts (MoE) inference. It jointly optimizes token compression, expert deployment, GPU-CPU residency, and routing to reduce cross-server communication and resource usage. The method uses a two-timescale alternating optimization, with an online loop compressing low-importance tokens and an offline loop updating expert placement based on accumulated traffic.

By Ning Li, Xinyu Wang, Xin Yuan, Wenchao Xu, Athanasios V. Vasilakos, Song Guo, Haijun Zhang

One email a morning, machine-written

Related stories

Fine-grained Computation-Communication Overlap via Tile-level Signaling and Scheduling for Mixture-of-Experts

Efficient Expert-Parallel Communication on PCIe-Connected Consumer GPUs

Topology-Aware Data Movement for Disaggregated GPU Inference

Mira: Memory-Efficient MoE Inference Using Adaptive Caching and Predictive Expert Staging

Achieving Cloud-Grade SLOs for Local Mixture-of-Experts Inference through CPU-GPU Hybrid Design

TopoCompress: Topology Aware Token Compression Algorithm for Distributed Edge MoE Inference