DropKV introduces a new KV‑cache eviction framework that decouples residual‑output perturbation into independent per‑token scores, avoiding the NP‑hard combinatorial optimization of prior methods. The approach offers a provable constant‑factor approximation under a cone condition verified on three long‑context LLMs, and a high‑performance Triton‑based implementation that yields a 19.8× speedup in scoring kernels and up to 9.9% end‑to‑end prefill speedup. Benchmarks on RULER and LongBench show DropKV consistently outperforms state‑of‑the‑art baselines at the same cache budgets, achieving the lowest inference latency among eviction strategies while remaining faster than dynamic cache baselines.
By Aozhong Zhang, Selcuk Gurses, Yanxia Deng, Naigang Wang, Chi-Chun Liu, Davis Wertheimer, Derrick Liu, Xin Li, Zi Yang, Felix X. -F. Ye, Penghang Yin
arXiv:2605. 25475v2 Announce Type: replace-cross Abstract: Large Language Models (LLMs) are increasingly expected to operate over long contexts, yet standard softmax attention incurs a KV cache that grows linearly with sequence length, quickly becoming the bottleneck for long context inference.
By Xintong Yang, Hao Gu, Binxing Xu, Lujun Li, Bei Liu, Jiacheng Liu, Qiyuan Zhu, Yike Guo, Sirui Han
arXiv:2606. 29563v1 Announce Type: cross Abstract: Large language models (LLMs) excel at complex tasks like question answering and summarization, thanks to their ability to handle long-context inputs.
By Shuvendu Roy, Mengyao Zhai, Hossein Hajimirsadeghi, Golnoosh Samei
arXiv:2606. 24467v1 Announce Type: new Abstract: Long-context large language model (LLM) inference is increasingly constrained by the memory footprint and decoding cost of key-value (KV) caches, limiting sustainable deployment on resource-constrained hardware.
By Xiaolin Lin, Jingcun Wang, Olga Kondrateva, Yiyu Shi, Bing Li, Grace Li Zhang
DropKV introduces a new key-value cache eviction strategy for large language models that decouples the joint eviction decision into independent per-token scoring, thereby avoiding the NP-hard combinatorial problem of minimizing output perturbation. The method offers a provable constant-factor approximation guarantee under a cone condition verified on three long-context LLMs, and its high-performance implementation using fused Triton kernels achieves a 19.8× speedup in scoring and up to 9.9% end-to-end prefill speedup. Evaluations on RULER and LongBench show that DropKV outperforms state-of-the-art baselines at equivalent cache budgets, delivering the lowest inference latency among eviction methods while remaining faster than the dynamic cache baseline.
Long-context large language model (LLM) inference is increasingly constrained by the memory footprint and decoding cost of key-value (KV) caches, limiting sustainable deployment on resource-constrained hardware. Existing KV cache eviction methods typically apply heuristic token scoring over all heads in GQA-based LLMs.