← Back to all news
arXiv Machine Learning By Jae Gon Kim, Donghoon Yoo, Hanyul Ryu, Sungho Ha, Juyeon Lee, Soojung Ryu

Phase-Decoupled, Model-Calibrated Power Control for Disaggregated LLM Serving

Read the original on arXiv Machine Learning →

The paper introduces a phase‑decoupled, model‑calibrated power controller for disaggregated large‑language‑model (LLM) serving, addressing the mismatch between GPU power settings and the distinct hardware regimes of prefill and decode stages. By calibrating separate power caps for each lane based on measured throughput‑latency cliffs, the authors achieve a 20.4% increase in tokens per joule with only a 3.5% rise in mean end‑to‑end latency on an 8‑node B200 cluster, outperforming NVIDIA’s Max‑Q profile. The approach also demonstrates consistent meeting of ITL‑p99 service‑level objectives across multiple MoE models and yields a 32.3% electricity savings over a sustained three‑day run.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

Hugging Face Trending Papers
Sep 10

Phase-Decoupled, Model-Calibrated Power Control for Disaggregated LLM Serving

The paper evaluates NVIDIA’s Max‑Q inference profile on a disaggregated B200 GPU system for large language model (LLM) serving, finding modest gains (+8.6% tokens/J) but increased latency (+5.2%). It proposes a phase‑decoupled, model‑calibrated power controller that sets a latency‑guaranteed SM‑clock window for prefill and a calibrated power cap for decode, achieving a 20.4% tokens/J improvement with only a 3.5% latency increase on an 8‑node Qwen3‑Coder‑480B deployment. The approach outperforms vendor profiles on both energy and latency, and demonstrates significant long‑term electricity savings in MoE‑based serving.

llmsagentsefficiency
More like this →
arXiv AI
Sep 16

Calibrate, Then Route: A Measured Study of Learned Request Routing for Disaggregated LLM Serving

The paper evaluates a learned request‑routing policy for disaggregated large‑language‑model serving, where compute‑heavy prefill and memory‑heavy decode stages run on separate GPU pools. Using a discrete‑event simulator and real NVIDIA A40 GPUs, the calibrated router—leveraging prompt length, predicted output length, KV‑cache pressure, and SLO class—outperforms round‑robin, least‑loaded, and length‑based heuristics, achieving the highest mean goodput (0.864) and lowest variance across three mixed, bursty arrival traces. Hardware calibration proves critical, providing a 4.5‑point goodput boost and roughly 40 % of the tail‑latency advantage, and the learned router can match round‑robin performance with one fewer GPU in certain scenarios.

By Srikanta Datta Tumkur, Jay Iyer, Mehar Simhadri, Sai Pavan Kumar, Sai Kapil Kumar, Ramesh Nampelly
llmsefficiency
More like this →