DropKV introduces a new KV‑cache eviction framework that decouples residual‑output perturbation into independent per‑token scores, avoiding the NP‑hard combinatorial optimization of prior methods. The approach offers a provable constant‑factor approximation under a cone condition verified on three long‑context LLMs, and a high‑performance Triton‑based implementation that yields a 19.8× speedup in scoring kernels and up to 9.9% end‑to‑end prefill speedup. Benchmarks on RULER and LongBench show DropKV consistently outperforms state‑of‑the‑art baselines at the same cache budgets, achieving the lowest inference latency among eviction strategies while remaining faster than dynamic cache baselines.
By Aozhong Zhang, Selcuk Gurses, Yanxia Deng, Naigang Wang, Chi-Chun Liu, Davis Wertheimer, Derrick Liu, Xin Li, Zi Yang, Felix X. -F. Ye, Penghang Yin
arXiv:2605. 25475v2 Announce Type: replace-cross Abstract: Large Language Models (LLMs) are increasingly expected to operate over long contexts, yet standard softmax attention incurs a KV cache that grows linearly with sequence length, quickly becoming the bottleneck for long context inference.
By Xintong Yang, Hao Gu, Binxing Xu, Lujun Li, Bei Liu, Jiacheng Liu, Qiyuan Zhu, Yike Guo, Sirui Han
arXiv:2606. 29563v1 Announce Type: cross Abstract: Large language models (LLMs) excel at complex tasks like question answering and summarization, thanks to their ability to handle long-context inputs.
By Shuvendu Roy, Mengyao Zhai, Hossein Hajimirsadeghi, Golnoosh Samei
arXiv:2606. 24467v1 Announce Type: new Abstract: Long-context large language model (LLM) inference is increasingly constrained by the memory footprint and decoding cost of key-value (KV) caches, limiting sustainable deployment on resource-constrained hardware.
By Xiaolin Lin, Jingcun Wang, Olga Kondrateva, Yiyu Shi, Bing Li, Grace Li Zhang
DropKV introduces a new key-value cache eviction strategy for large language models that decouples the joint eviction decision into independent per-token scoring, thereby avoiding the NP-hard combinatorial problem of minimizing output perturbation. The method offers a provable constant-factor approximation guarantee under a cone condition verified on three long-context LLMs, and its high-performance implementation using fused Triton kernels achieves a 19.8× speedup in scoring and up to 9.9% end-to-end prefill speedup. Evaluations on RULER and LongBench show that DropKV outperforms state-of-the-art baselines at equivalent cache budgets, delivering the lowest inference latency among eviction methods while remaining faster than the dynamic cache baseline.
Long-context large language model (LLM) inference is increasingly constrained by the memory footprint and decoding cost of key-value (KV) caches, limiting sustainable deployment on resource-constrained hardware. Existing KV cache eviction methods typically apply heuristic token scoring over all heads in GQA-based LLMs.
arXiv:2607. 27600v1 Announce Type: new Abstract: Key-value (KV) cache management through compression and eviction strategies has emerged as an important research direction in recent years.
By Stephen Gould, Anton van den Hengel
arXiv:2602. 10238v2 Announce Type: replace-cross Abstract: The growing size of Large Language Models (LLMs) makes efficient inference challenging, primarily due to the memory demands of the autoregressive Key-Value (KV) cache.
By Luca Moschella, Laura Manduchi, Ozan Sener
The paper introduces EpiKV, an epiphany‑aware key–value cache eviction strategy that avoids using the attention matrix. It leverages hidden‑state shifts and recent query–key relevance to rank cached tokens, matching or surpassing the performance of existing attention‑based eviction methods while remaining compatible with fast inference kernels. Experiments on multiple benchmarks show that EpiKV improves inference throughput without sacrificing accuracy.
By Steven Kolawole, Virginia Smith
arXiv:2605.23258v2 Announce Type: replace
Abstract: KV cache growth is a major bottleneck for long-context inference in large language models. Existing methods are often dominated by binary eviction...
By Yuping Lin, Jiayuan Ding, Yue Xing, Pengfei He, Jiliang Tang, Subhabrata Mukherjee
arXiv:2609.39329v1 Announce Type: new
Abstract: Long-context inference with Large Language Models (LLMs) is bottlenecked by the linearly growing memory of the key-value (KV) cache. Existing compressi...
By Chanryeol Lee, Chanhyuk Lee, Yeonwoo Choi, Donggyun Kim, Seunghoon Hong
arXiv:2606. 01563v1 Announce Type: new Abstract: Autoregressive decoding in Transformer-based language models relies on the KV cache, whose memory footprint grows linearly with sequence length and becomes the primary bottleneck for long-context inference.
By Yu Li, Binxu Li, Tian Lan