arXiv AI
Jul 24
Error Certificates for KV-Cache Eviction via Randomized Design
arXiv:2607. 21475v1 Announce Type: cross Abstract: Deterministic KV-cache eviction keeps the top-$k$ tokens under an importance score and deletes the rest.
Risk-Controlled KV-Cache Eviction: From Memory Budgets to Risk Targets
Related stories
Hugging Face Trending Papers
Jul 23
Error Certificates for KV-Cache Eviction via Randomized Design
Deterministic KV-cache eviction keeps the top-$k$ tokens under an importance score and deletes the rest. We prove that this design cannot know what it destroyed: evicted values can be altered so that everything the serving system retains is unchanged while the true attention-output error grows arbitrarily, so no serving-time estimator of that error is consistent.
arXiv Machine Learning
Sep 22
PAGE: Partition-Aware Gated KV-Cache Eviction
PAGE is a partition‑aware gated KV‑cache eviction method that reframes eviction as a per‑input admission decision. It uses a single label‑free scalar— the early‑to‑late drop in pairwise top‑k head agreement—to classify inputs into a capacity‑bound class (where eviction is catastrophic) and a dilution‑prone class (where eviction is safe or beneficial). By thresholding this drop, PAGE applies a base evictor only when necessary, reducing the harm rate in the capacity‑bound regime from 0.75 to 0.026 and achieving a 29× improvement across four models and benchmarks without retraining the evictor.
arXiv AI
Jul 22
CompilerKV: Risk-Adaptive KV Compression via Offline Experience Compilation
arXiv:2602. 08686v3 Announce Type: replace-cross Abstract: Prefill-only KV compression freezes a token subset at the end of prefill and decodes from it without further eviction.
arXiv AI
Jun 17
AnchorKV: Safety-Aware KV Cache Compression via Soft Penalty with a Refusal Anchor
arXiv:2606. 17872v1 Announce Type: cross Abstract: Large language models (LLMs) outperform earlier architectures on generative inference and long-context tasks, but their large size introduces significant challenges in memory usage, energy cost, and on-device deployment.
arXiv Computation and Language
Sep 10
KVShareArena: KV-Cache Reuse Across Contexts and Model Checkpoints
arXiv:2609.10266v1 Announce Type: new Abstract: LLM serving systems already reuse KV caches, but only when the reused text sits at the very start of the prompt. Two growing workloads break this condi...
arXiv Machine Learning
Aug 4
RestoreKV: Recovering Full-Cache Behavior Under Aggressive Query-Agnostic KV Cache Eviction
arXiv:2608. 01247v1 Announce Type: cross Abstract: Query-agnostic KV cache eviction compresses a context once and reuses the resulting cache for arbitrary future queries, but performance can collapse under tight budgets.
arXiv AI
Aug 11
RippleKV: Cross-Layer KV Cache Allocation via Perturbation Propagation
arXiv:2608. 08684v1 Announce Type: cross Abstract: Long-context LLM inference is bottlenecked by KV cache memory, yet distributing a limited cache budget across layers remains challenging.
arXiv AI
Aug 11
Governing the KV Cache: Preventing Timing Side-Channel Leakage in Multi-Tenant LLM Inference
arXiv:2608. 09225v1 Announce Type: cross Abstract: The key-value (KV) cache is the primary throughput optimization in modern large language model (LLM) inference, enabling prefix reuse across requests.
Hugging Face Trending Papers
Aug 2
RestoreKV: Recovering Full-Cache Behavior Under Aggressive Query-Agnostic KV Cache Eviction
Query-agnostic KV cache eviction compresses a context once and reuses the resulting cache for arbitrary future queries, but performance can collapse under tight budgets. Existing methods primarily improve which original KV pairs are retained.
arXiv AI
Aug 7
Runtime Observability for Heterogeneous Attention Memory
arXiv:2608. 05863v1 Announce Type: new Abstract: Modern models no longer keep a plain KV cache: latent caches, learned sparse selectors and recurrent states each carry the model's memory in a different form, and each fails differently under compression.
arXiv AI
Sep 18
Exploring a Layer-Wise Design Space for KV Cache Eviction
The paper investigates whether key‑value (KV) cache eviction strategies should vary across Transformer layers. By combining existing eviction methods in different layer configurations and profiling their performance, the authors find that heterogeneous, layer‑wise routing consistently outperforms homogeneous policies on LongBench tasks. Even with a fixed set of methods, the placement of each method strongly influences overall quality, and a single well‑chosen route surpasses all nine standalone baselines across multiple cache budgets.