The paper investigates when attention maps can be compressed beyond simple sparsity, arguing that large weights alone do not guarantee compressibility. It introduces metrics such as global score gaps and weighted sums of omitted values to determine token retention for a target mass, and presents a retrieval–aggregation model to predict the impact of truncation. Based on these insights, the authors propose CertKV, a training‑free compressor that allocates a tail‑summary slot per head and distributes remaining slots according to value dispersion, achieving strong performance across several benchmarks.
By Chiwun Yang, Xiaoyu Li
The paper introduces Random Attention, a method that evicts KV cache entries uniformly at random within each attention head while preserving the prompt. Experiments on four models and six reasoning tasks show that this simple strategy matches the performance of the best existing eviction methods and achieves 32‑43% higher throughput in vLLM deployments. The authors explain that the prompt is the most fragile cache component and that reasoning traces are redundantly stored across text and attention heads, making a selection score unnecessary.
By Heng Wang, Jielin Qiu, Wenting Zhao, Cheng Qian, Liangwei Yang, Jiawei Han, Heng Ji, Silvio Savarese, Shelby Heinecke, Huan Wang
arXiv:2607. 15498v1 Announce Type: cross Abstract: The key-value (KV) cache is the main memory bottleneck in long-context large language model (LLM) inference.
By Shahrzad Esmat, Dhawal Shah, Ali Jannesari
ValueDiff introduces a value‑geometric KV cache eviction strategy for large language models that suppress attention sinks. It ranks tokens by the L2 deviation of their value vectors from the cache mean, a score that aligns with minimal‑disturbance eviction under a max‑entropy assumption. Across several benchmarks—RULER, LongBench, and MATH‑500—ValueDiff consistently retains a higher proportion of useful tokens than prior methods, especially under tight cache budgets.
By Junyoung Park, Jungwook Choi, Mingu Lee
arXiv:2609.30738v1 Announce Type: new
Abstract: KV cache eviction methods such as SnapKV and PyramidKV rank tokens solely by mean attention over a small observation window. We study a unified score,...
By Tianfang Xie, Wei Zhu
The paper introduces Random Attention, a method that evicts KV cache entries uniformly at random within each attention head while preserving the prompt. It demonstrates that this simple strategy matches or surpasses more complex eviction schemes across four models and six reasoning tasks, achieving 32‑43% higher throughput in vLLM deployments. Experiments reveal that the prompt is the most fragile cache component and that redundancy in the reasoning trace across text and attention heads protects against random eviction, eliminating the need for a selection score.