arXiv AI
Sep 15

mKernel: Fast Multi-GPU, Multi-Node Fused Kernels

arXiv:2609.13585v1 Announce Type: cross Abstract: Communication has become a bottleneck in distributed training and inference of large models. Overlapping communication with computation at the granul...

By Ziming Mao, Yihan Zhang, Shawn Wei Chew, Shuang Ma, Costin Raiciu, Yang Zhou, Scott Shenker, Ion Stoica
arXiv Machine Learning
6d ago

One-Sided Expert-Parallel All-to-All for Mixture-of-Experts Inference over NVLink

The paper introduces NVLinkOneSided, a one‑sided expert‑parallel all‑to‑all communication scheme for Mixture‑of‑Experts inference on GPUs connected via NVLink. By pushing tokens directly into target GPU buffers without metadata exchange and using grouped GEMMs to read them in place, the method reduces fixed‑cost latency, especially at small batch sizes. Two synchronization protocols—Fence for high throughput and Sentinel for low latency—enable the approach, which achieves 1.2–3.0× faster dispatch and combine than DeepEP and up to 2.4× faster MoE layers on Blackwell GPUs, outperforming DeepGEMM MegaMoE up to 128 tokens per GPU. The implementation is modular, agnostic to activation quantization, and easily integrable into inference frameworks.

By Bo Li, Chulian Zhang, Or Biri, Anthony Chang, Anton Korzh, Noam Duvdevani, XingFei Xi, Daniel Stokes, Zhenhuan Chen, Ludwig Schneider, Artem Y. Polyakov, Julien Demouth, Chandler Zhou
arXiv AI
Sep 28

Scepsy: Serving Agentic Workflows Using Aggregate LLM Pipelines

Scepsy is a serving system designed to efficiently schedule arbitrary multi‑LLM agentic workflows on GPU clusters. It leverages the observation that each LLM’s share of execution time remains relatively stable across requests, profiling LLMs under various parallelism levels to build an Aggregate LLM Pipeline that predicts throughput and latency. Using this predictor, Scepsy searches for optimal GPU allocations—balancing fractional GPU shares, tensor parallelism, and replica counts—and then heuristically places them on the cluster to reduce fragmentation and honor network topology, achieving up to 2.5× higher throughput and 1.0–3.3× lower latency compared to baseline approaches.

By Otto White, Marcel Wagenl\"ander, Britannio Jarrett, Xijin Zhao, Yanda Tao, Pedro Silvestre, Guo Li, Huanzhou Zhu, Llu\'is Vilanova, Peter Pietzuch