The paper introduces Random Attention, a method that evicts KV cache entries uniformly at random within each attention head while preserving the prompt. It demonstrates that this simple strategy matches or surpasses more complex eviction schemes across four models and six reasoning tasks, achieving 32‑43% higher throughput in vLLM deployments. Experiments reveal that the prompt is the most fragile cache component and that redundancy in the reasoning trace across text and attention heads protects against random eviction, eliminating the need for a selection score.
The paper introduces EpiKV, an epiphany‑aware key–value cache eviction strategy that avoids using the attention matrix. It leverages hidden‑state shifts and recent query–key relevance to rank cached tokens, matching or surpassing the performance of existing attention‑based eviction methods while remaining compatible with fast inference kernels. Experiments on multiple benchmarks show that EpiKV improves inference throughput without sacrificing accuracy.
By Steven Kolawole, Virginia Smith
arXiv:2606. 26472v1 Announce Type: new Abstract: As reasoning models emit chains of thought tens of thousands of tokens long, KV cache increasingly becomes a deployment bottleneck.
By Steven Kolawole, Virginia Smith
arXiv:2607. 05061v1 Announce Type: new Abstract: Key-value (KV) cache growth is a major bottleneck in autoregressive decoding, as memory and bandwidth scale linearly with context length.
By Lukas Hauzenberger, Niklas Schmidinger, Anamaria-Roberta Hartl, David Stap, Thomas Schmied, Sebastian B\"ock, G\"unter Klambauer, Sepp Hochreiter
arXiv:2606. 23961v1 Announce Type: new Abstract: Long-context and agentic LLM workloads push the KV cache past any fixed memory budget, forcing the inference stack to permanently evict tokens at every step of a continuous-inference stream.
By Duc Duong, Hoang Anh Duy Le, Jianwen Xie, Anshumali Shrivastava, Zhaozhuo Xu
arXiv:2606. 03928v1 Announce Type: new Abstract: Reasoning models improve accuracy through extended chains of thought, but their long outputs create a memory and compute bottleneck.
By Ting-Yun Chang, Harvey Yiyun Fu, Deqing Fu, Chenghao Yang, Jesse Thomason, Robin Jia
The paper introduces Draft‑Guided Eviction (DGE), a training‑free KV‑cache compression technique that postpones token eviction until after drafting the first two answer tokens, thereby preserving the cache entries needed for the answer trajectory. Unlike prior methods that focus on optimizing what to keep, DGE changes the timing of eviction, allowing existing compression schemes such as SnapKV, H2O, and PyramidKV to be applied without altering their eviction scores. Experiments show DGE consistently outperforms previous approaches across multiple instruct‑tuned models, achieving near‑full‑cache performance on LongBench and demonstrating that the improvement stems from trajectory anchoring rather than changes in cache selection.
By Haeyong Kang, Chang D. Yoo
The paper investigates KV‑cache eviction strategies for sparse‑attention models, showing that selecting the largest attention weights is nearly optimal—closing only a median 2–5 % of the gap to full attention. It further demonstrates that differences in performance between eviction methods largely stem from memory usage, with the new training‑free ContourKV allocator outperforming state‑of‑the‑art methods in most pairwise comparisons while matching their byte‑efficiency.
By Jack Shi, Jerry Gu
arXiv:2607. 10582v1 Announce Type: cross Abstract: Large language model (LLM) agents accumulate heterogeneous context, including system instructions, plans, user turns, retrieved documents, tool outputs, and intermediate reasoning, whose key-value (KV) cache can become a major memory bottleneck.
By Venkatesha Matam, Keon Kim
arXiv:2606. 29563v1 Announce Type: cross Abstract: Large language models (LLMs) excel at complex tasks like question answering and summarization, thanks to their ability to handle long-context inputs.
By Shuvendu Roy, Mengyao Zhai, Hossein Hajimirsadeghi, Golnoosh Samei
arXiv:2602. 03203v2 Announce Type: replace-cross Abstract: Recently, large language models (LLMs) have shown remarkable reasoning abilities by producing long reasoning traces.
By Zican Dong, Peiyu Liu, Junyi Li, Zhipeng Chen, Han Peng, Shuo Wang, Wayne Xin Zhao
arXiv:2610.05685v2 Announce Type: replace
Abstract: Reasoning models write most of their KV cache while decoding long chains of thought (CoT), so the cache has to be compressed online under a fixed m...
By Runguo Li