arXiv AI
PowerSlider: Exploiting Phase Asymmetry for LLM Serving under Demand Response
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
Hugging Face Trending Papers
Sep 10
Phase-Decoupled, Model-Calibrated Power Control for Disaggregated LLM Serving
The paper evaluates NVIDIA’s Max‑Q inference profile on a disaggregated B200 GPU system for large language model (LLM) serving, finding modest gains (+8.6% tokens/J) but increased latency (+5.2%). It proposes a phase‑decoupled, model‑calibrated power controller that sets a latency‑guaranteed SM‑clock window for prefill and a calibrated power cap for decode, achieving a 20.4% tokens/J improvement with only a 3.5% latency increase on an 8‑node Qwen3‑Coder‑480B deployment. The approach outperforms vendor profiles on both energy and latency, and demonstrates significant long‑term electricity savings in MoE‑based serving.
arXiv Machine Learning
Sep 11
Phase-Decoupled, Model-Calibrated Power Control for Disaggregated LLM Serving
The paper introduces a phase‑decoupled, model‑calibrated power controller for disaggregated large‑language‑model (LLM) serving, addressing the mismatch between GPU power settings and the distinct hardware regimes of prefill and decode stages. By calibrating separate power caps for each lane based on measured throughput‑latency cliffs, the authors achieve a 20.4% increase in tokens per joule with only a 3.5% rise in mean end‑to‑end latency on an 8‑node B200 cluster, outperforming NVIDIA’s Max‑Q profile. The approach also demonstrates consistent meeting of ITL‑p99 service‑level objectives across multiple MoE models and yields a 32.3% electricity savings over a sustained three‑day run.
arXiv AI
Aug 14
TEMPO: Makespan-Aware Expert-Parallel Load Balancing Across Memory- and Compute-Bound Regimes
arXiv:2608. 13057v1 Announce Type: cross Abstract: In expert-parallel (EP) MoE serving, every layer synchronizes at the slowest GPU.
arXiv Machine Learning
Oct 16
When the KV Cache Is Slower Than the GPU: Scaling Remote KV Tiers for Context-Parallel Prefill
The paper investigates the performance impact of using a remote KV cache tier for long‑context language model serving. In a disaggregated deployment of GLM‑5.3 on 8xB300 nodes, enabling the remote tier initially increased the 16k‑token p99 latency from 3.96 s to 17.3 s, and synchronous loading caused severe queue instability. By sharding the KV store across context‑parallel ranks, partitioning the remote tier, adding asynchronous loading, and fixing client bugs, the authors reduced the p99 latency from 83.32 s to 0.67 s and increased prefill throughput by 14–34 %. whyItMatters":"The study demonstrates that careful sharding and asynchronous handling of remote KV caches can dramatically improve latency and throughput in large‑scale LLM serving systems."
arXiv AI
Jul 3
Towards Load-Aware Prefill Deflection for Disaggregated LLM Serving
arXiv:2607. 02043v1 Announce Type: cross Abstract: Disaggregated LLM serving runs prefill and decode on separate GPU pools to keep the two phases from interfering.
arXiv Machine Learning
Jun 2
Observation, Not Prediction: Conversation-Level Disaggregated Scheduling for Agentic Serving
arXiv:2606. 01839v1 Announce Type: cross Abstract: LLM-based agents resolve a user task through many turns of dependent inference and tool calls, producing a workload whose total cost is unknown when the task arrives.