arXiv AI By Ishaan Datta, Alind Khare, Koushik Kumar Dey, Ratnesh Jamidar, Anand Kumar Singh

VelServe: Taming Decode Jitters in LLM Inference with Tail-Calibrated Scheduling

Read the original on arXiv AI →

VelServe is a scheduler designed to reduce decode jitters in large language model (LLM) inference, ensuring that latency service-level objectives (SLOs) for time to first token (TTFT) and time between tokens (TBT) are met. It calibrates the amount of prefill work each decode step performs based on the measured tail latency of that step, leaving headroom to prevent overruns and prioritizing shorter waiting prompts. Experiments on multiple attention families and workloads show that VelServe improves SLO goodput by 2.4–2.9× over the best baseline on Qwen3‑8B and DeepSeek‑V4‑Flash, and by at least 4.4× on DeepSeek‑V2‑Lite, while meeting SLOs for more than twice as many requests when prefill and decode are disaggregated across eight GPUs.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv Machine Learning
Aug 10

Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving

arXiv:2608. 06557v1 Announce Type: cross Abstract: The reasoning and agentic capabilities of large language models have expanded the range of applications they support, from short interactive exchanges to long, compute-heavy requests.

By Muhammad Adnan, Rohan Mahapatra, Prashant J. Nair, Daniel Berger, Pantea Zardoshti, Rodrigo Fonseca, Esha Choukse
arXiv AI
Oct 7

FluidPD: In-Place Elasticity for SLO-Aware Prefill-Decode Disaggregated LLM Serving

FluidPD is a prefill‑decode disaggregated LLM serving system that introduces in‑place elasticity to meet SLOs. It uses FluidToken to offload a bounded portion of prefill work to decode workers during short‑term imbalance, and FluidRole to reassign running workers between prefill and decode roles for sustained imbalance, all guided by lightweight pressure indices. In production Azure trace workloads, FluidPD improves overall SLO attainment over static SGLang by up to 94.6 percentage points without adding workers.

By Kartik Ramesh, Kaidi Fu, Zihan Zheng, Jiahuan Yu, Fabio Oliveira, Carlos Costa, Minjia Zhang