The paper presents py‑kvcache, a new KV offload connector for vLLM that uses asynchronous direct I/O, bounded shared staging, and scheduler‑aware preloading to improve external KV caching performance on NVMe SSDs. Experiments across synthetic workloads, long‑context benchmarks, and production traces show that py‑kvcache can load 80k‑token prefixes 2.0× faster than LMCache, with preloading contributing an additional 1.34× speedup, and achieves overall performance within 4% of native vLLM KV Offload. The study highlights that cache effectiveness depends on transfer granularity, intermediate memory use, and scheduling timing rather than just device bandwidth, indicating that external KV caching should be considered a setup‑specific admission decision.
By Joseph Kanichai, Tiziano De Matteis, Animesh Trivedi
arXiv:2607. 02630v1 Announce Type: cross Abstract: Hardware accelerators now sit on the critical path of online serving.
By Bojie Li
The paper evaluates a learned request‑routing policy for disaggregated large‑language‑model serving, where compute‑heavy prefill and memory‑heavy decode stages run on separate GPU pools. Using a discrete‑event simulator and real NVIDIA A40 GPUs, the calibrated router—leveraging prompt length, predicted output length, KV‑cache pressure, and SLO class—outperforms round‑robin, least‑loaded, and length‑based heuristics, achieving the highest mean goodput (0.864) and lowest variance across three mixed, bursty arrival traces. Hardware calibration proves critical, providing a 4.5‑point goodput boost and roughly 40 % of the tail‑latency advantage, and the learned router can match round‑robin performance with one fewer GPU in certain scenarios.
By Srikanta Datta Tumkur, Jay Iyer, Mehar Simhadri, Sai Pavan Kumar, Sai Kapil Kumar, Ramesh Nampelly
Crossflow introduces an elastic boundary for prefilling and decoding in large language model serving, allowing decode nodes to publish short‑lived leases that limit prefilling resources and output projections. By adapting to dynamic phase demand, Crossflow improves token throughput by 16.2‑17.4% on average and up to 43.4% under high load, while consistently reducing mean time‑to‑first‑token. The approach eliminates the inefficiencies of static partitioning, which can leave 17% of cluster capacity idle or cause queueing and lost throughput.
By Yi Xu, Ehsan K. Ardestani, Wenyin Fu, Martin Schatz, Krishna Malladi, Zhan Shu, Adnan Aziz, Shobhit Kanaujia, Ajit Mathews, Chunqiang Tang
The paper introduces an elastic key‑value (KV) cache for large language model (LLM) serving that dynamically reclaims a pre‑allocated reserve during decode‑heavy phases and restores it before prefill, using a userspace CUDA virtual‑memory trick that requires no driver changes. The authors implement this mechanism, test it under realistic workloads, and find that it offers only marginal benefits—about a 1 % difference in time‑to‑first‑token for large prefill chunks—and that simpler strategies such as lowering the maximum batch size can achieve similar results. The study also notes that the reserve’s impact diminishes with higher tensor‑parallelism levels.
whyItMatters":"The work demonstrates that a dynamic KV cache reclamation strategy can be implemented without driver patches and that its practical benefits are limited, guiding future LLM serving optimizations toward simpler approaches."
By Sathishkumar Sivashanmugam
Diffusion language models (dLLMs) generate text by iteratively denoising a masked response and can commit multiple output positions per model invocation. Their bidirectional attention prevents exact autoregressive-style KV caching, since committing one position shifts the KV activations of all others.
arXiv:2607. 04206v1 Announce Type: cross Abstract: Diffusion language models (dLLMs) generate text by iteratively denoising a masked response and can commit multiple output positions per model invocation.
By Nitin Kedia, Saurabh Agarwal, Myungjin Lee, Aditya Akella
arXiv:2606. 01839v1 Announce Type: cross Abstract: LLM-based agents resolve a user task through many turns of dependent inference and tool calls, producing a workload whose total cost is unknown when the task arrives.
By Jianru Ding, Ryien Hosseini, Pouya Mahdi Gholami, Mingyuan Xiang, Henry Hoffmann
arXiv:2609.16215v1 Announce Type: new
Abstract: GPU high bandwidth memory is scarce and expensive, and KV caches consume much of it as chats, agent loops, and document question answering accumulate s...
By Srikanta Datta Tumkur, Jay Iyer, Mehar Simhadri, Sai Pavan Kumar, Sai Kapil Kumar, Ramesh Nampelly
WiSP (Working‑Set Paging) is a routing‑aware expert pager that allows Mixture‑of‑Experts models to run on GPUs that cannot hold the entire expert pool by paging experts in and out of VRAM while preserving byte‑identical outputs. On a 24 GiB RTX 3090, WiSP doubles decode throughput compared to static offload when the model does not fit, and its companion policy MV‑WSA allocates VRAM between resident experts and KV cache based on marginal latency benefit, reducing end‑to‑end time by up to 1.19× without altering model outputs.
By Jiamu Zhang, Liang Wu, Mayank Darbari, Liangjie Hong
arXiv:2606. 03910v1 Announce Type: cross Abstract: Disaggregated LLM inference forces the KV cache to traverse the datacenter network before decoding begins, so transfer time enters directly into the Time to First Token (TTFT) budget.
By Mubarak Adetunji Ojewale
arXiv:2606. 21633v2 Announce Type: replace Abstract: The KV cache dominates GPU memory in long-context LLM serving, crowding out batch capacity and leaving GPU compute idle.
By Omin Kwon, Doyeon Kim, Jongseok Park, Seung Yul Lee, Ion Stoica, Jae W. Lee