The paper investigates how temporal aggregation and ranking preservation affect aggressive decoding-time KV cache eviction. It shows that exponential‑moving‑average (EMA) aggregation makes many scorer modifications indistinguishable at the eviction‑set level, while certain variants (KeyDiff, key norm, recency, learned scorer) alter rankings and degrade performance. Building on this, the authors propose InertiaKV and its lazy variant, achieving 1.34–1.46× faster decoding, and also explore a score‑free decoding approach that slightly improves quality while eliminating further scoring.
By Bo Zeng, Yu Zhao, Yefeng Liu, Zhihong Lu, Xuanfan Ni, Xintong Wang
arXiv:2610.06927v1 Announce Type: cross
Abstract: The key-value (KV) cache of autoregressive transformers grows linearly with context length and dominates memory at long context. Most training-free r...
By Sara Abdali, Jongwoo Ko, Pashmina Cameron
NestedKV is a training‑free key‑only KV cache compression technique for long‑context language models that uses global, block‑level, and sliding‑window key anchors to score tokens via multi‑time‑scale cosine anomaly. It combines these rankings with a head‑adaptive outer learner and surprise‑gated token routing, requiring no model modification or additional training. Experiments on Qwen3 and Llama‑3.2 across benchmarks such as RULER, LongBench, and MMLU‑Pro show that NestedKV outperforms existing methods when the retained cache is small, achieving up to 19‑point gains on RULER and LongBench at a 75% retention rate.
By Hong Chen, Xiang Liu, Yubo Gao, Yuxuan Fan, Bo Wang, Yuanlin Chu, Yuanguo Lin, Xuming Hu
arXiv:2605.23258v2 Announce Type: replace
Abstract: KV cache growth is a major bottleneck for long-context inference in large language models. Existing methods are often dominated by binary eviction...
By Yuping Lin, Jiayuan Ding, Yue Xing, Pengfei He, Jiliang Tang, Subhabrata Mukherjee
arXiv:2607. 15498v1 Announce Type: cross Abstract: The key-value (KV) cache is the main memory bottleneck in long-context large language model (LLM) inference.
By Shahrzad Esmat, Dhawal Shah, Ali Jannesari
The paper introduces Draft‑Guided Eviction (DGE), a training‑free KV‑cache compression technique that postpones token eviction until after drafting the first two answer tokens, thereby preserving the cache entries needed for the answer trajectory. Unlike prior methods that focus on optimizing what to keep, DGE changes the timing of eviction, allowing existing compression schemes such as SnapKV, H2O, and PyramidKV to be applied without altering their eviction scores. Experiments show DGE consistently outperforms previous approaches across multiple instruct‑tuned models, achieving near‑full‑cache performance on LongBench and demonstrating that the improvement stems from trajectory anchoring rather than changes in cache selection.
By Haeyong Kang, Chang D. Yoo