← Back to all news
arXiv AI By Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, Xinran Xu

Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving

Read the original on arXiv AI →

Mooncake is a serving platform for Kimi, a leading LLM service by Moonshot AI, that uses a KVCache‑centric disaggregated architecture to separate prefill and decoding clusters. It leverages underutilized CPU, DRAM, and SSD resources in the GPU cluster to create a disaggregated KVCache and employs a scheduler that balances throughput and latency SLOs. To handle highly overloaded scenarios, Mooncake introduces a prediction‑based early rejection policy, achieving up to a 525% throughput increase in simulations and enabling Kimi to process 75% more requests under real workloads.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Oct 9

SparKV: Overhead-Aware KV Cache Loading for Efficient On-Device LLM Inference

SparKV is an adaptive framework for on-device large language model inference that intelligently decides whether to stream or compute each key-value (KV) cache chunk, overlapping communication and computation to cut latency. It dynamically adjusts offline-generated schedules at runtime to cope with changing wireless connectivity and edge resource availability. Experiments show SparKV can reduce Time-to-First-Token by 1.3×–5.1× and lower per-request energy consumption by 1.5×–3.3× with minimal impact on response quality.

By Hongyao Liu, Liuqun Zhai, Junyi Wang, Zhengru Fang, Jingshu Chen, Jun Huang
llmsefficiency
More like this →
arXiv AI
Sep 16

Calibrate, Then Route: A Measured Study of Learned Request Routing for Disaggregated LLM Serving

The paper evaluates a learned request‑routing policy for disaggregated large‑language‑model serving, where compute‑heavy prefill and memory‑heavy decode stages run on separate GPU pools. Using a discrete‑event simulator and real NVIDIA A40 GPUs, the calibrated router—leveraging prompt length, predicted output length, KV‑cache pressure, and SLO class—outperforms round‑robin, least‑loaded, and length‑based heuristics, achieving the highest mean goodput (0.864) and lowest variance across three mixed, bursty arrival traces. Hardware calibration proves critical, providing a 4.5‑point goodput boost and roughly 40 % of the tail‑latency advantage, and the learned router can match round‑robin performance with one fewer GPU in certain scenarios.

By Srikanta Datta Tumkur, Jay Iyer, Mehar Simhadri, Sai Pavan Kumar, Sai Kapil Kumar, Ramesh Nampelly
llmsefficiency
More like this →