arXiv AI
Jul 24
Error Certificates for KV-Cache Eviction via Randomized Design
arXiv:2607. 21475v1 Announce Type: cross Abstract: Deterministic KV-cache eviction keeps the top-$k$ tokens under an importance score and deletes the rest.
A Probabilistic Interpretation of KV Cache Eviction
Related stories
Hugging Face Trending Papers
Jul 23
Error Certificates for KV-Cache Eviction via Randomized Design
Deterministic KV-cache eviction keeps the top-$k$ tokens under an importance score and deletes the rest. We prove that this design cannot know what it destroyed: evicted values can be altered so that everything the serving system retains is unchanged while the true attention-output error grows arbitrarily, so no serving-time estimator of that error is consistent.
arXiv Machine Learning
Jul 3
The risk of KV cache compression
arXiv:2607. 01520v1 Announce Type: new Abstract: Transformer inference on long sequences is expensive because softmax attention repeatedly reads from a large KV cache.
arXiv AI
Aug 11
Rethinking KV Cache Eviction via a Unified Information-Theoretic Objective
arXiv:2604. 25975v2 Announce Type: replace-cross Abstract: Key-Value (KV) caching is essential for large language model inference, yet its memory overhead poses a critical bottleneck for long-context generation.
arXiv Machine Learning
Jun 2
MomentKV: Closing the Directional Gap in KV Cache Eviction for Long-Context Inference
arXiv:2606. 01563v1 Announce Type: new Abstract: Autoregressive decoding in Transformer-based language models relies on the KV cache, whose memory footprint grows linearly with sequence length and becomes the primary bottleneck for long-context inference.
arXiv Computation and Language
Sep 4
Random Attention: Rethinking KV Cache Eviction for Efficient Reasoning
The paper introduces Random Attention, a method that evicts KV cache entries uniformly at random within each attention head while preserving the prompt. Experiments on four models and six reasoning tasks show that this simple strategy matches the performance of the best existing eviction methods and achieves 32‑43% higher throughput in vLLM deployments. The authors explain that the prompt is the most fragile cache component and that reasoning traces are redundantly stored across text and attention heads, making a selection score unnecessary.
arXiv AI
Aug 18
KV-Rescue: Recovering Reasoning Language Model KV Eviction Loss via Stepwise Interleaving
arXiv:2608. 15797v1 Announce Type: new Abstract: KV-cache eviction caps the memory cost of long reasoning traces but is inherently lossy because the model decodes from a partial view of its history.
arXiv Machine Learning
Sep 24
Risk-Controlled KV-Cache Eviction: From Memory Budgets to Risk Targets
arXiv:2609.27981v1 Announce Type: cross Abstract: KV-cache eviction is typically evaluated through average quality-memory trade-offs, yet a small average loss can hide requests whose utility degrades...
arXiv Machine Learning
Aug 11
CommitKV: Lifecycle-Aware KV Cache Compression via Commit Transitions for Multi-Turn Agents
arXiv:2608. 07855v1 Announce Type: new Abstract: Multi-turn Reasoning-and-Acting (ReAct) agents accumulate growing trajectories of reasoning, tool calls, and observations.
arXiv AI
Aug 6
Fewer Tokens, Smaller Cache: Reward-Coordinated Efficient Reasoning
arXiv:2608. 04771v1 Announce Type: new Abstract: Large Reasoning Models (LRMs) excel on complex tasks through long chain-of-thought (CoT) reasoning, but their lengthy intermediate steps cause severe overthinking that inflates inference cost.
arXiv Machine Learning
Aug 4
RestoreKV: Recovering Full-Cache Behavior Under Aggressive Query-Agnostic KV Cache Eviction
arXiv:2608. 01247v1 Announce Type: cross Abstract: Query-agnostic KV cache eviction compresses a context once and reuses the resulting cache for arbitrary future queries, but performance can collapse under tight budgets.
Hugging Face Trending Papers
Sep 3
Random Attention: Rethinking KV Cache Eviction for Efficient Reasoning
The paper introduces Random Attention, a method that evicts KV cache entries uniformly at random within each attention head while preserving the prompt. It demonstrates that this simple strategy matches or surpasses more complex eviction schemes across four models and six reasoning tasks, achieving 32‑43% higher throughput in vLLM deployments. Experiments reveal that the prompt is the most fragile cache component and that redundancy in the reasoning trace across text and attention heads protects against random eviction, eliminating the need for a selection score.