arXiv Computation and Language
The Gating Mechanism of KV Cache Eviction
Read the original on arXiv Computation and Language →The Flow has not summarised this story yet — read it at arXiv Computation and Language.
Related stories
arXiv AI
Oct 15
GateSight: Recurrent Memory Anticipates Attention Memory for Query-Agnostic KV Eviction in Hybrid LLMs
arXiv:2610.14754v1 Announce Type: new Abstract: Hybrid language models interleave Gated DeltaNet (GDN) layers, whose recurrent memory has a fixed size, with full-attention layers, whose key-value (KV...
arXiv Machine Learning
Sep 22
ValueDiff: Value-Geometric KV Cache Eviction for Sink-Suppressed LLMs
ValueDiff introduces a value‑geometric KV cache eviction strategy for large language models that suppress attention sinks. It ranks tokens by the L2 deviation of their value vectors from the cache mean, a score that aligns with minimal‑disturbance eviction under a max‑entropy assumption. Across several benchmarks—RULER, LongBench, and MATH‑500—ValueDiff consistently retains a higher proportion of useful tokens than prior methods, especially under tight cache budgets.
arXiv Computation and Language
Sep 4
Random Attention: Rethinking KV Cache Eviction for Efficient Reasoning
The paper introduces Random Attention, a method that evicts KV cache entries uniformly at random within each attention head while preserving the prompt. Experiments on four models and six reasoning tasks show that this simple strategy matches the performance of the best existing eviction methods and achieves 32‑43% higher throughput in vLLM deployments. The authors explain that the prompt is the most fragile cache component and that reasoning traces are redundantly stored across text and attention heads, making a selection score unnecessary.
arXiv Machine Learning
Sep 22
PAGE: Partition-Aware Gated KV-Cache Eviction
PAGE is a partition‑aware gated KV‑cache eviction method that reframes eviction as a per‑input admission decision. It uses a single label‑free scalar— the early‑to‑late drop in pairwise top‑k head agreement—to classify inputs into a capacity‑bound class (where eviction is catastrophic) and a dilution‑prone class (where eviction is safe or beneficial). By thresholding this drop, PAGE applies a base evictor only when necessary, reducing the harm rate in the capacity‑bound regime from 0.75 to 0.026 and achieving a 29× improvement across four models and benchmarks without retraining the evictor.
Hugging Face Trending Papers
Sep 3
Random Attention: Rethinking KV Cache Eviction for Efficient Reasoning
The paper introduces Random Attention, a method that evicts KV cache entries uniformly at random within each attention head while preserving the prompt. It demonstrates that this simple strategy matches or surpasses more complex eviction schemes across four models and six reasoning tasks, achieving 32‑43% higher throughput in vLLM deployments. Experiments reveal that the prompt is the most fragile cache component and that redundancy in the reasoning trace across text and attention heads protects against random eviction, eliminating the need for a selection score.
arXiv AI
Aug 25
Sigmoid Attention as a Better Substrate for Learned KV Cache Eviction
arXiv:2608.23296v1 Announce Type: cross Abstract: Learned KV-cache eviction often faces a soft-to-hard mismatch: during training, differentiable gates typically attenuate token contributions, whereas...