HeadWiseKV is a training‑free framework that compresses the residual global key–value caches of hybrid long‑context language models by assigning each physical KV head a static, multilevel history window. It formulates cache allocation as a restricted operational rate–distortion problem and uses the SeqCalib algorithm to generate per‑head residency policies that account for interactions across layers. In evaluations on four hybrid models, HeadWiseKV preserves near‑full‑KV quality while reducing peak device memory usage by 8.59% at a 112K context length and extending the largest verified context from 114K to 161K.
By Renjie Xie, Juncheng Yang, Aoting Hu, Mingxi Zhang, Liyao Wu, Zheheng Hong, Wei Xu
arXiv:2606. 02964v1 Announce Type: cross Abstract: Large Language Model (LLM) inference relies on key-value (KV) caches to avoid redundant attention computation.
By Chunan Shi, Yilei Chen, Yilin Chen, Xupeng Miao, Bin Cui
arXiv:2606. 06256v1 Announce Type: new Abstract: As the input length of large language model (LLM) serving continues to grow, the KV cache has become a dominant bottleneck in AI infrastructure.
By Yang Liu, ZhaoKai Luo, HuaYi Jin, ZhiYong Wang, RuoZhou He, BoYu Wang, Guanjie Chen, Junhao Hu
Long-context large language model (LLM) inference is increasingly constrained by the memory footprint and decoding cost of key-value (KV) caches, limiting sustainable deployment on resource-constrained hardware. Existing KV cache eviction methods typically apply heuristic token scoring over all heads in GQA-based LLMs.
arXiv:2606. 24467v1 Announce Type: new Abstract: Long-context large language model (LLM) inference is increasingly constrained by the memory footprint and decoding cost of key-value (KV) caches, limiting sustainable deployment on resource-constrained hardware.
By Xiaolin Lin, Jingcun Wang, Olga Kondrateva, Yiyu Shi, Bing Li, Grace Li Zhang
Large language models are increasingly deployed with persistent personalized context, such as accumulated memory profiles or long conversation histories, that is shared across a user's many requests. Production memory systems (e.
arXiv:2607. 27090v1 Announce Type: cross Abstract: Large language models are increasingly deployed with persistent personalized context, such as accumulated memory profiles or long conversation histories, that is shared across a user's many requests.
By Peter Li, Prashant Pandey
Multi-turn Large Language Model (LLM) serving is critical for consistent user experiences, yet the linear growth of the Key-Value (KV) cache imposes significant pressure on GPU memory and bandwidth. Non-uniform KV compression effectively preserves more information by considering the individual importance of each KV cache.
arXiv:2606. 26666v1 Announce Type: new Abstract: Autoregressive large language model (LLM) serving is increasingly limited by key-value (KV) cache movement rather than dense matrix multiplication.
By Muhammad Ahmed
arXiv:2606. 06302v1 Announce Type: new Abstract: Multi-turn Large Language Model (LLM) serving is critical for consistent user experiences, yet the linear growth of the Key-Value (KV) cache imposes significant pressure on GPU memory and bandwidth.
By Hyungmin Kim, Minsoo Kim, Hongseok Kim, Jungwook Choi
arXiv:2604. 26968v2 Announce Type: replace-cross Abstract: Key-value (KV) cache memory management is the primary bottleneck limiting throughput and cost-efficiency in large-scale GPU inference serving.
By Sanjeev Rao Ganjihal
arXiv:2608.30963v1 Announce Type: cross
Abstract: Modern large language model (LLM) serving systems increasingly operate over repeated or shared context, yet each model typically performs its own pre...
By Yi Li, Dongming Jiang, Yi Zhao, Bingzhe Li