Hugging Face Trending Papers
One Megakernel is Not Enough: Predicting Where to Cut for Streaming Inference
Read the original on Hugging Face Trending Papers →The Flow has not summarised this story yet — read it at Hugging Face Trending Papers.
Related stories
arXiv Machine Learning
4d ago
One Megakernel is Not Enough: Predicting Where to Cut for Streaming Inference
The paper investigates how to optimally partition streaming inference models into megakernels and conventional graph components. It introduces a framework that predicts the best megakernel composition without full end‑to‑end runs, showing that multiple megakernels or a fully conventional graph often outperform a single megakernel across various models and batch sizes. The study defines four model categories based on megakernel stability and demonstrates significant latency and throughput gains on an H100 NVL GPU using a three‑megakernel design.
arXiv AI
Jun 30
KernelSight-LM: A Kernel-Level LLM Inference Simulator
arXiv:2606. 28565v1 Announce Type: cross Abstract: As large language models (LLMs) move into production serving, practitioners must rapidly evaluate inference performance across diverse hardware, models, and serving parameters to meet cost and latency targets.
arXiv AI
Sep 15
mKernel: Fast Multi-GPU, Multi-Node Fused Kernels
arXiv:2609.13585v1 Announce Type: cross Abstract: Communication has become a bottleneck in distributed training and inference of large models. Overlapping communication with computation at the granul...
arXiv Machine Learning
Jun 11
MPK: A Compiler and Runtime for Mega-Kernelizing Tensor Programs
arXiv:2512. 22219v2 Announce Type: replace-cross Abstract: We introduce Mirage Persistent Kernel (MPK), the first compiler and runtime system that automatically transforms multi-GPU model inference into a single high-performance mega-kernel.
arXiv AI
Jul 28
X-Stage: An Overlooked Pipeline Stage for Communication-Computation Overlap in DiT Inference
arXiv:2607. 23264v1 Announce Type: cross Abstract: Fine-grained, device-initiated communication lets persistent GPU kernels in distributed diffusion transformer (DiT) inference issue remote stores and overlap data movement with Tensor Core computation.
arXiv AI
Jul 23
Fine-grained Computation-Communication Overlap via Tile-level Signaling and Scheduling for Mixture-of-Experts
arXiv:2607. 19539v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) architectures increase model capacity without proportionally increasing computation cost and have become a key building block for scaling large language models (LLMs) to trillion-parameter regimes.