arXiv:2607. 02043v1 Announce Type: cross Abstract: Disaggregated LLM serving runs prefill and decode on separate GPU pools to keep the two phases from interfering.
By Shrikara Arun, Anjaly Parayil, Srikant Bharadwaj, Renee St. Amant, Victor R\"uhle
GroupKV is a lightweight hierarchical KV cache management system designed for long‑context diffusion large language model (dLLM) inference. It partitions the context into contiguous groups and uses coarse‑to‑fine sparse selection, cross‑layer consistency for predictive prefetching, and a staleness correction mechanism to keep the cache coherent amid dynamic KV updates. The approach also incorporates streaming prefill to lower peak memory usage, achieving up to 48× longer serviceable context, 3.73× faster inference in offload‑based settings, and competitive task accuracy.
By Jinhao Wang, Zhexin Hu, Kangjie Zhou, Xin Zhou, Fangfang Liu
The paper presents py‑kvcache, a new KV offload connector for vLLM that uses asynchronous direct I/O, bounded shared staging, and scheduler‑aware preloading to improve external KV caching performance on NVMe SSDs. Experiments across synthetic workloads, long‑context benchmarks, and production traces show that py‑kvcache can load 80k‑token prefixes 2.0× faster than LMCache, with preloading contributing an additional 1.34× speedup, and achieves overall performance within 4% of native vLLM KV Offload. The study highlights that cache effectiveness depends on transfer granularity, intermediate memory use, and scheduling timing rather than just device bandwidth, indicating that external KV caching should be considered a setup‑specific admission decision.
By Joseph Kanichai, Tiziano De Matteis, Animesh Trivedi
arXiv:2606. 21633v2 Announce Type: replace Abstract: The KV cache dominates GPU memory in long-context LLM serving, crowding out batch capacity and leaving GPU compute idle.
By Omin Kwon, Doyeon Kim, Jongseok Park, Seung Yul Lee, Ion Stoica, Jae W. Lee
arXiv:2607. 02525v1 Announce Type: cross Abstract: We present PEEK, a lightweight scheduling and eviction framework for both online (streaming) and offline (batch) LLM serving; this paper focuses on the online regime.
By Bing Xie, Zhipeng Wang, Masahiro Tanaka, Zheng Zhen
Serving a 235B-parameter Mixture-of-Experts (MoE) model on a single 8 GB GPU is bottlenecked not by compute but by memory bandwidth: decode must stream each token's active experts from whichever tier holds them, and on consumer hardware most experts sit on an SSD far slower than RAM. We quantify this bandwidth wall on Qwen3-235B (Q4_K_M, 134 GB): measured decode is 0.