The paper investigates KV‑cache eviction strategies for sparse‑attention models, showing that selecting the largest attention weights is nearly optimal—closing only a median 2–5 % of the gap to full attention. It further demonstrates that differences in performance between eviction methods largely stem from memory usage, with the new training‑free ContourKV allocator outperforming state‑of‑the‑art methods in most pairwise comparisons while matching their byte‑efficiency.
By Jack Shi, Jerry Gu
Long-context large language model (LLM) inference is increasingly constrained by the memory footprint and decoding cost of key-value (KV) caches, limiting sustainable deployment on resource-constrained hardware. Existing KV cache eviction methods typically apply heuristic token scoring over all heads in GQA-based LLMs.
The paper introduces Draft‑Guided Eviction (DGE), a training‑free KV‑cache compression technique that postpones token eviction until after drafting the first two answer tokens, thereby preserving the cache entries needed for the answer trajectory. Unlike prior methods that focus on optimizing what to keep, DGE changes the timing of eviction, allowing existing compression schemes such as SnapKV, H2O, and PyramidKV to be applied without altering their eviction scores. Experiments show DGE consistently outperforms previous approaches across multiple instruct‑tuned models, achieving near‑full‑cache performance on LongBench and demonstrating that the improvement stems from trajectory anchoring rather than changes in cache selection.
By Haeyong Kang, Chang D. Yoo
ChampionKV introduces an input‑adaptive, head‑wise budget allocator for KV cache compression, using the maximum aggregated observation‑window attention score as a lightweight proxy for attention concentration. It assigns larger budgets to heads with dispersed attention and smaller budgets to concentrated heads, and incorporates an abnormal‑head mask and floor‑budget mechanism to enhance robustness. When integrated with existing compressors like SnapKV and PyramidKV, ChampionKV consistently improves performance on LongBench and RULER, especially on Llama3.1‑70B‑Instruct under tight budgets, while maintaining competitive results on smaller models.
By Yuechuan Chen, Yanda Guo, Xiaoxi Liu, Weilin Ruan, Huanbo Xu, Ruyi Wang, Wei Zhu
arXiv:2606. 24467v1 Announce Type: new Abstract: Long-context large language model (LLM) inference is increasingly constrained by the memory footprint and decoding cost of key-value (KV) caches, limiting sustainable deployment on resource-constrained hardware.
By Xiaolin Lin, Jingcun Wang, Olga Kondrateva, Yiyu Shi, Bing Li, Grace Li Zhang
arXiv:2608.27128v4 Announce Type: replace
Abstract: Compressing a key-value cache before its next question is known requires choosing what to retain without knowing which evidence will matter. Value...
By Hong Chen, Yudong Zeng, Yongwei Huang, Zuhao Ouyang, Dongnan Zheng, Junyan Zhang, Yubo Gao, Xuming Hu
arXiv:2609.30738v1 Announce Type: new
Abstract: KV cache eviction methods such as SnapKV and PyramidKV rank tokens solely by mean attention over a small observation window. We study a unified score,...
By Tianfang Xie, Wei Zhu
arXiv:2610.06927v1 Announce Type: cross
Abstract: The key-value (KV) cache of autoregressive transformers grows linearly with context length and dominates memory at long context. Most training-free r...
By Sara Abdali, Jongwoo Ko, Pashmina Cameron
arXiv:2607. 06519v1 Announce Type: new Abstract: Long-context LLM inference is increasingly limited by the memory and bandwidth cost of KV caches, yet aggressive compression can remove the layer-specific evidence needed for retrieval and multi-step reasoning.
By Anna C\'ordoba, Adam Puente Tercero, Nerea Angulo Hijo, Mar Linares Tercero, Julia Barrientos, Ainhoa Miranda, Jes\'us Olivera
EchoPress is a training‑free method for pruning key‑value caches in large language models. It approximates the reconstruction attention used by KVzip by leveraging queries and keys from standard prefill, reconstructing only the first chunk to calibrate importance scores for the rest of the context. Experiments on LongBench and RULER with Qwen3‑8B and Llama‑3.1‑8B‑Instruct show that EchoPress matches KVzip’s task accuracy across eviction ratios from 50% to 90%, while reducing compression overhead by 1.7–19.6× and total prefill time by up to 2.9×.
By Jiawei Lin, Saibo Geng, Thomas Bourgeat
KV$^2$ is a query‑agnostic key‑value cache compression technique that selectively reconstructs only informative in‑context tokens using a lightweight proxy scorer before final eviction scoring. On benchmarks such as RULER, Needle‑in‑a‑Haystack, and LongBench, KV$^2$ outperforms baseline methods, especially under tight memory budgets, achieving higher scores with lower runtime and peak memory than full‑context reconstruction. The approach demonstrates that reusable KV‑cache compression can avoid reprocessing the entire prompt while maintaining quality.
By Johannes Wesch, Danni Liu, Jan Niehues
arXiv:2610.08811v1 Announce Type: new
Abstract: As context windows scale to tens or hundreds of thousands of tokens, KV cache compression has become essential for efficient LLM inference. Existing me...
By Linfeng Dong