arXiv AI By Huihao Jing, Wenbin Hu, Shaojin Chen, Haochen Shi, Haoyu Huang, Sirui Zhang, Yuxuan Liu, Zhongwei Xie, Haoran Li

RadixFlow: Filling Execution Bubbles in Prefix-Cached Hybrid LLM Serving

Read the original on arXiv AI →

The Flow has not summarised this story yet — read it at arXiv AI.

arXiv AI
6d ago

Bursts That Outlast Themselves: Persistent Degradation in Prefix-Cached Agent Serving

The paper investigates how prefix caching in large language model (LLM) agents can lead to persistent degradation when small bursts of additional requests occur. Even with identical base workloads, a brief surge can push a vLLM server into a degraded state—reducing hit ratios, throughput, and causing a backlog that grows for over an hour—while a server without the burst remains healthy. The study identifies that slower service keeps more sessions active, expanding the working set beyond the key‑value pool and triggering repeated recomputations, and demonstrates that simple mitigation strategies such as admission throttling, session suspension, and a budgeter that limits resident KV demand can prevent or recover from this collapse.

By Wen Huang, Jack Liu
arXiv AI
1d ago

VelServe: Taming Decode Jitters in LLM Inference with Tail-Calibrated Scheduling

VelServe is a scheduler designed to reduce decode jitters in large language model (LLM) inference, ensuring that latency service-level objectives (SLOs) for time to first token (TTFT) and time between tokens (TBT) are met. It calibrates the amount of prefill work each decode step performs based on the measured tail latency of that step, leaving headroom to prevent overruns and prioritizing shorter waiting prompts. Experiments on multiple attention families and workloads show that VelServe improves SLO goodput by 2.4–2.9× over the best baseline on Qwen3‑8B and DeepSeek‑V4‑Flash, and by at least 4.4× on DeepSeek‑V2‑Lite, while meeting SLOs for more than twice as many requests when prefill and decode are disaggregated across eight GPUs.

By Ishaan Datta, Alind Khare, Koushik Kumar Dey, Ratnesh Jamidar, Anand Kumar Singh
Hugging Face Trending Papers
Nov 10

Bursts That Outlast Themselves: Persistent Degradation in Prefix-Cached Agent Serving

The paper investigates how prefix caching in LLM agent serving can lead to persistent degradation when traffic bursts occur. It shows that even a small increase in arrivals can push a vLLM server into a low hit‑ratio, low‑throughput regime that continues to deteriorate for an extended period, while a server without prefix caching remains stable. The authors identify a workload‑balance condition that predicts this collapse and evaluate mitigation strategies such as throttling, session suspension, and a KV‑budgeting approach, finding the latter most effective.

arXiv Computation and Language
Nov 4

Pacing Branch Parallelism in LLM Serving

The paper introduces PACE, a per‑step admission controller that manages intra‑request parallelism in large language model serving. By admitting extra decoding branches only when the predicted branch externality fits within the batch’s current slack, PACE avoids the pitfalls of eager or fixed‑cap branch admission. Experiments on Qwen3‑32B and other large models show that PACE boosts goodput by up to 2.15× while maintaining 99% SLO attainment.

By Swapnil Gandhi, Siva Kumar Sastry Hari, William J. Dally, Christos Kozyrakis