ValueDiff introduces a value‑geometric KV cache eviction strategy for large language models that suppress attention sinks. It ranks tokens by the L2 deviation of their value vectors from the cache mean, a score that aligns with minimal‑disturbance eviction under a max‑entropy assumption. Across several benchmarks—RULER, LongBench, and MATH‑500—ValueDiff consistently retains a higher proportion of useful tokens than prior methods, especially under tight cache budgets.
arXiv:2611.07605v1 Announce Type: new Abstract: KV cache eviction must decide which entries carry information that the rest of the cache can no longer supply. A natural way to estimate this is a lear...
arXiv:2610.14754v1 Announce Type: new Abstract: Hybrid language models interleave Gated DeltaNet (GDN) layers, whose recurrent memory has a fixed size, with full-attention layers, whose key-value (KV...
arXiv:2610.07643v1 Announce Type: cross Abstract: Most KV-cache eviction methods ask, in effect, which memory appeared important while reading the prompt? We instead ask, which memory will matter whi...
The paper investigates KV‑cache eviction strategies for sparse‑attention models, showing that selecting the largest attention weights is nearly optimal—closing only a median 2–5 % of the gap to full attention. It further demonstrates that differences in performance between eviction methods largely stem from memory usage, with the new training‑free ContourKV allocator outperforming state‑of‑the‑art methods in most pairwise comparisons while matching their byte‑efficiency.
arXiv:2607. 05061v1 Announce Type: new Abstract: Key-value (KV) cache growth is a major bottleneck in autoregressive decoding, as memory and bandwidth scale linearly with context length.
arXiv:2606. 26472v1 Announce Type: new Abstract: As reasoning models emit chains of thought tens of thousands of tokens long, KV cache increasingly becomes a deployment bottleneck.
arXiv:2610.06996v1 Announce Type: cross Abstract: Block diffusion language models keep a large key-value (KV) cache throughout generation and attend to it at every denoising step, limiting both memor...
The paper introduces Draft‑Guided Eviction (DGE), a training‑free KV‑cache compression technique that postpones token eviction until after drafting the first two answer tokens, thereby preserving the cache entries needed for the answer trajectory. Unlike prior methods that focus on optimizing what to keep, DGE changes the timing of eviction, allowing existing compression schemes such as SnapKV, H2O, and PyramidKV to be applied without altering their eviction scores. Experiments show DGE consistently outperforms previous approaches across multiple instruct‑tuned models, achieving near‑full‑cache performance on LongBench and demonstrating that the improvement stems from trajectory anchoring rather than changes in cache selection.
arXiv:2608. 05326v1 Announce Type: new Abstract: Autoregressive large language model inference is increasingly constrained by the memory footprint of the Key-Value (KV) cache.
The paper introduces EpiKV, an epiphany‑aware key–value cache eviction strategy that avoids using the attention matrix. It leverages hidden‑state shifts and recent query–key relevance to rank cached tokens, matching or surpassing the performance of existing attention‑based eviction methods while remaining compatible with fast inference kernels. Experiments on multiple benchmarks show that EpiKV improves inference throughput without sacrificing accuracy.
arXiv:2610.06927v1 Announce Type: cross Abstract: The key-value (KV) cache of autoregressive transformers grows linearly with context length and dominates memory at long context. Most training-free r...
arXiv:2606. 01563v1 Announce Type: new Abstract: Autoregressive decoding in Transformer-based language models relies on the KV cache, whose memory footprint grows linearly with sequence length and becomes the primary bottleneck for long-context inference.