The paper investigates when attention maps can be compressed beyond simple sparsity, arguing that large weights alone do not guarantee compressibility. It introduces metrics such as global score gaps and weighted sums of omitted values to determine token retention for a target mass, and presents a retrieval–aggregation model to predict the impact of truncation. Based on these insights, the authors propose CertKV, a training‑free compressor that allocates a tail‑summary slot per head and distributes remaining slots according to value dispersion, achieving strong performance across several benchmarks.
By Chiwun Yang, Xiaoyu Li
The paper introduces Random Attention, a method that evicts KV cache entries uniformly at random within each attention head while preserving the prompt. Experiments on four models and six reasoning tasks show that this simple strategy matches the performance of the best existing eviction methods and achieves 32‑43% higher throughput in vLLM deployments. The authors explain that the prompt is the most fragile cache component and that reasoning traces are redundantly stored across text and attention heads, making a selection score unnecessary.
By Heng Wang, Jielin Qiu, Wenting Zhao, Cheng Qian, Liangwei Yang, Jiawei Han, Heng Ji, Silvio Savarese, Shelby Heinecke, Huan Wang
arXiv:2607. 15498v1 Announce Type: cross Abstract: The key-value (KV) cache is the main memory bottleneck in long-context large language model (LLM) inference.
By Shahrzad Esmat, Dhawal Shah, Ali Jannesari
The paper introduces Random Attention, a method that evicts KV cache entries uniformly at random within each attention head while preserving the prompt. It demonstrates that this simple strategy matches or surpasses more complex eviction schemes across four models and six reasoning tasks, achieving 32‑43% higher throughput in vLLM deployments. Experiments reveal that the prompt is the most fragile cache component and that redundancy in the reasoning trace across text and attention heads protects against random eviction, eliminating the need for a selection score.
arXiv:2608. 01247v1 Announce Type: cross Abstract: Query-agnostic KV cache eviction compresses a context once and reuses the resulting cache for arbitrary future queries, but performance can collapse under tight budgets.
By Changwoo Baek, Seungjun Shin, Kyeongbo Kong
arXiv:2607. 11942v1 Announce Type: cross Abstract: KV-cache compression methods are predominantly evaluated with the query appended to the context before compression -- a query-aware protocol.
By Daming Luo, Christy Liang, Junyu Xuan