The paper investigates how temporal aggregation of token scores during decoding-time KV cache compression affects eviction decisions. It shows that using an exponential‑moving‑average (EMA) aggregation keeps ranking changes minimal for many scorer variants, while others like KeyDiff or recency significantly alter rankings and degrade performance. Building on this, the authors propose InertiaKV and its lazy variant, which achieve 1.34–1.46× faster decoding than full refresh, and also evaluate a score‑free approach that slightly improves quality while eliminating further scoring.
NestedKV is a training‑free key‑only KV cache compression technique for long‑context language models that uses global, block‑level, and sliding‑window key anchors to score tokens via multi‑time‑scale cosine anomaly. It combines these rankings with a head‑adaptive outer learner and surprise‑gated token routing, requiring no model modification or additional training. Experiments on Qwen3 and Llama‑3.2 across benchmarks such as RULER, LongBench, and MMLU‑Pro show that NestedKV outperforms existing methods when the retained cache is small, achieving up to 19‑point gains on RULER and LongBench at a 75% retention rate.
By Hong Chen, Xiang Liu, Yubo Gao, Yuxuan Fan, Bo Wang, Yuanlin Chu, Yuanguo Lin, Xuming Hu
arXiv:2607. 15498v1 Announce Type: cross Abstract: The key-value (KV) cache is the main memory bottleneck in long-context large language model (LLM) inference.
By Shahrzad Esmat, Dhawal Shah, Ali Jannesari
arXiv:2608.23843v1 Announce Type: new
Abstract: Long-context inference in large language models (LLMs) is increasingly limited by the memory required for the key-value (KV) cache. KV cache compressio...
By Zizhong Wang, Jieying Wang, Zhao Zhang, Jiajia Li
arXiv:2602. 10238v2 Announce Type: replace-cross Abstract: The growing size of Large Language Models (LLMs) makes efficient inference challenging, primarily due to the memory demands of the autoregressive Key-Value (KV) cache.
By Luca Moschella, Laura Manduchi, Ozan Sener
arXiv:2608. 05326v1 Announce Type: new Abstract: Autoregressive large language model inference is increasingly constrained by the memory footprint of the Key-Value (KV) cache.
By Ayushman Garg, Akshita Gupta, Shaswata Bhattacharya, Abhishek Gupta, Sandeep Kumar, Manoj Kumar