arXiv:2609.06853v1 Announce Type: cross
Abstract: Shared key--value (KV) cache reuse improves large language model (LLM) serving, but it can also create a timing side channel that reveals whether a p...
By Rana Abu Bakar
arXiv:2608.07911v4 Announce Type: replace
Abstract: Mixture-of-Experts (MoE) models have outgrown accelerator memory, and offloading expert weights to host memory is now standard. This makes expert c...
By Yu Zhang
arXiv:2608. 07911v1 Announce Type: new Abstract: Mixture-of-Experts (MoE) models have outgrown accelerator memory, and offloading expert weights to host memory is now standard.
By Yu Zhang
arXiv:2607. 10389v1 Announce Type: cross Abstract: A persistent interactive world model keeps its running state resident on the GPU that serves it: a multi-gigabyte attention cache, almost all of it rewritten at every generation step.
By Jin Li (Harvard University), Jiawei Chen
The paper introduces an elastic key‑value (KV) cache for large language model (LLM) serving that dynamically reclaims a pre‑allocated reserve during decode‑heavy phases and restores it before prefill, using a userspace CUDA virtual‑memory trick that requires no driver changes. The authors implement this mechanism, test it under realistic workloads, and find that it offers only marginal benefits—about a 1 % difference in time‑to‑first‑token for large prefill chunks—and that simpler strategies such as lowering the maximum batch size can achieve similar results. The study also notes that the reserve’s impact diminishes with higher tensor‑parallelism levels.
whyItMatters":"The work demonstrates that a dynamic KV cache reclamation strategy can be implemented without driver patches and that its practical benefits are limited, guiding future LLM serving optimizations toward simpler approaches."
By Sathishkumar Sivashanmugam
arXiv:2609.37626v1 Announce Type: cross
Abstract: No single way of parallelizing attention serves large language models well under all loads. Low concurrency favors tensor parallelism, many independe...
By Chuan Liu, Shuoming Zhang, Zhicheng Li, Qianqi Sun, Ruiyuan Xu, Qiuchu Yu, Xiyu Shi, Huimin Cui, Jiacheng Zhao