arXiv:2607. 02525v1 Announce Type: cross Abstract: We present PEEK, a lightweight scheduling and eviction framework for both online (streaming) and offline (batch) LLM serving; this paper focuses on the online regime.
By Bing Xie, Zhipeng Wang, Masahiro Tanaka, Zheng Zhen
The paper investigates how prefix caching in large language model (LLM) agents can lead to persistent degradation when small bursts of additional requests occur. Even with identical base workloads, a brief surge can push a vLLM server into a degraded state—reducing hit ratios, throughput, and causing a backlog that grows for over an hour—while a server without the burst remains healthy. The study identifies that slower service keeps more sessions active, expanding the working set beyond the key‑value pool and triggering repeated recomputations, and demonstrates that simple mitigation strategies such as admission throttling, session suspension, and a budgeter that limits resident KV demand can prevent or recover from this collapse.
By Wen Huang, Jack Liu
VelServe is a scheduler designed to reduce decode jitters in large language model (LLM) inference, ensuring that latency service-level objectives (SLOs) for time to first token (TTFT) and time between tokens (TBT) are met. It calibrates the amount of prefill work each decode step performs based on the measured tail latency of that step, leaving headroom to prevent overruns and prioritizing shorter waiting prompts. Experiments on multiple attention families and workloads show that VelServe improves SLO goodput by 2.4–2.9× over the best baseline on Qwen3‑8B and DeepSeek‑V4‑Flash, and by at least 4.4× on DeepSeek‑V2‑Lite, while meeting SLOs for more than twice as many requests when prefill and decode are disaggregated across eight GPUs.
By Ishaan Datta, Alind Khare, Koushik Kumar Dey, Ratnesh Jamidar, Anand Kumar Singh
The paper investigates how prefix caching in LLM agent serving can lead to persistent degradation when traffic bursts occur. It shows that even a small increase in arrivals can push a vLLM server into a low hit‑ratio, low‑throughput regime that continues to deteriorate for an extended period, while a server without prefix caching remains stable. The authors identify a workload‑balance condition that predicts this collapse and evaluate mitigation strategies such as throttling, session suspension, and a KV‑budgeting approach, finding the latter most effective.
arXiv:2609.37626v1 Announce Type: cross
Abstract: No single way of parallelizing attention serves large language models well under all loads. Low concurrency favors tensor parallelism, many independe...
By Chuan Liu, Shuoming Zhang, Zhicheng Li, Qianqi Sun, Ruiyuan Xu, Qiuchu Yu, Xiyu Shi, Huimin Cui, Jiacheng Zhao
The paper introduces PACE, a per‑step admission controller that manages intra‑request parallelism in large language model serving. By admitting extra decoding branches only when the predicted branch externality fits within the batch’s current slack, PACE avoids the pitfalls of eager or fixed‑cap branch admission. Experiments on Qwen3‑32B and other large models show that PACE boosts goodput by up to 2.15× while maintaining 99% SLO attainment.
By Swapnil Gandhi, Siva Kumar Sastry Hari, William J. Dally, Christos Kozyrakis
The paper introduces an elastic key‑value (KV) cache for large language model (LLM) serving that dynamically reclaims a pre‑allocated reserve during decode‑heavy phases and restores it before prefill, using a userspace CUDA virtual‑memory trick that requires no driver changes. The authors implement this mechanism, test it under realistic workloads, and find that it offers only marginal benefits—about a 1 % difference in time‑to‑first‑token for large prefill chunks—and that simpler strategies such as lowering the maximum batch size can achieve similar results. The study also notes that the reserve’s impact diminishes with higher tensor‑parallelism levels.
whyItMatters":"The work demonstrates that a dynamic KV cache reclamation strategy can be implemented without driver patches and that its practical benefits are limited, guiding future LLM serving optimizations toward simpler approaches."
By Sathishkumar Sivashanmugam
TOPAS is a Task‑Oriented Prefix‑Aware Scheduler designed for multi‑agent large language model serving. It jointly decides which agent prefixes to retain in a shared key‑value cache and which requests to schedule, balancing the reduction of each task’s longest remaining service path against the benefit of downstream prefix reuse while accounting for movement and preemption costs. Experiments on synthetic DAGs and MetaGPT software‑development workflows show that TOPAS can reduce mean and p99 job completion times by up to 39.8%/49.4% and 22.0%/26.6% respectively compared to the best baselines.
By Hongqiu Ni, Han Tian, Chi Zhang, Guopeng Li, Haisheng Tan
The study evaluates how configuration choices, particularly chunked prefill, affect performance differences between two large language model serving engines, vLLM and SGLang. Using a reproducible Kubernetes testbed and a seeded multi‑agent workload, the authors show that matching the chunked prefill setting eliminates a large portion of the observed capacity and latency gaps. Even after matching, vLLM still exhibits a tail‑latency advantage due to cache‑aware request ordering, and the authors provide six recommendations for fair engine comparisons.
By Muhammad Faizan Anwar, Sina Keshvadi
arXiv:2607. 02043v1 Announce Type: cross Abstract: Disaggregated LLM serving runs prefill and decode on separate GPU pools to keep the two phases from interfering.
By Shrikara Arun, Anjaly Parayil, Srikant Bharadwaj, Renee St. Amant, Victor R\"uhle
The paper investigates the performance impact of using a remote KV cache tier for long‑context language model serving. In a disaggregated deployment of GLM‑5.3 on 8xB300 nodes, enabling the remote tier initially increased the 16k‑token p99 latency from 3.96 s to 17.3 s, and synchronous loading caused severe queue instability. By sharding the KV store across context‑parallel ranks, partitioning the remote tier, adding asynchronous loading, and fixing client bugs, the authors reduced the p99 latency from 83.32 s to 0.67 s and increased prefill throughput by 14–34 %.
whyItMatters":"The study demonstrates that careful sharding and asynchronous handling of remote KV caches can dramatically improve latency and throughput in large‑scale LLM serving systems."
By Muhammad Ibrahim
Diffusion language models (dLLMs) generate text by iteratively denoising a masked response and can commit multiple output positions per model invocation. Their bidirectional attention prevents exact autoregressive-style KV caching, since committing one position shifts the KV activations of all others.