arXiv AI
Monte Carlo Estimation for KV Cache Eviction
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Jul 7
KVpop -- Key-Value Cache Compression with Predictive Online Pruning
arXiv:2607. 05061v1 Announce Type: new Abstract: Key-value (KV) cache growth is a major bottleneck in autoregressive decoding, as memory and bandwidth scale linearly with context length.
arXiv Machine Learning
Sep 22
ValueDiff: Value-Geometric KV Cache Eviction for Sink-Suppressed LLMs
ValueDiff introduces a value‑geometric KV cache eviction strategy for large language models that suppress attention sinks. It ranks tokens by the L2 deviation of their value vectors from the cache mean, a score that aligns with minimal‑disturbance eviction under a max‑entropy assumption. Across several benchmarks—RULER, LongBench, and MATH‑500—ValueDiff consistently retains a higher proportion of useful tokens than prior methods, especially under tight cache budgets.
arXiv Machine Learning
Jun 2
MomentKV: Closing the Directional Gap in KV Cache Eviction for Long-Context Inference
arXiv:2606. 01563v1 Announce Type: new Abstract: Autoregressive decoding in Transformer-based language models relies on the KV cache, whose memory footprint grows linearly with sequence length and becomes the primary bottleneck for long-context inference.
arXiv Machine Learning
Jun 29
Learning to Evict from Key-Value Cache
arXiv:2602. 10238v2 Announce Type: replace-cross Abstract: The growing size of Large Language Models (LLMs) makes efficient inference challenging, primarily due to the memory demands of the autoregressive Key-Value (KV) cache.
arXiv Machine Learning
Jul 31
Back from the Future: Key-Value Cache Management by Counter-Causal Surprise
arXiv:2607. 27600v1 Announce Type: new Abstract: Key-value (KV) cache management through compression and eviction strategies has emerged as an important research direction in recent years.
arXiv AI
Aug 11
DistillCache: KL-Guided Adaptive KV-Cache Eviction for Memory-Efficient LLM Inference
arXiv:2608. 08878v1 Announce Type: cross Abstract: Transformer-based large language models (LLMs) achieve strong performance across many tasks, but their Key-Value (KV) cache grows linearly with sequence length, creating a severe memory bottleneck for long-context inference.