arXiv AI
AvoKV-E: Payload-Aware KV Cache Eviction for Long Reasoning
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
6d ago
EpiKV: Epiphany-Aware KV Cache Eviction Without the Attention Matrix
The paper introduces EpiKV, an epiphany‑aware key–value cache eviction strategy that avoids using the attention matrix. It leverages hidden‑state shifts and recent query–key relevance to rank cached tokens, matching or surpassing the performance of existing attention‑based eviction methods while remaining compatible with fast inference kernels. Experiments on multiple benchmarks show that EpiKV improves inference throughput without sacrificing accuracy.
arXiv Machine Learning
Sep 22
StepKV: Step-Aware KV Cache Compression for LLM Agents
StepKV introduces a step-aware approach to compressing the key-value cache used during large language model inference, treating reasoning steps as primary units of retention rather than individual tokens. By linking cache entries to the steps that generated them and estimating each step’s utility from trajectory signals, StepKV assigns a combined token‑ and step‑level score to guide pruning. Experiments on multi‑hop question answering and long‑horizon web reasoning show that StepKV maintains accuracy even under tight cache budgets, outperforming token‑level baselines that suffer sharp performance drops.
arXiv Machine Learning
Jun 29
Learning to Evict from Key-Value Cache
arXiv:2602. 10238v2 Announce Type: replace-cross Abstract: The growing size of Large Language Models (LLMs) makes efficient inference challenging, primarily due to the memory demands of the autoregressive Key-Value (KV) cache.
arXiv Machine Learning
Jul 7
KVpop -- Key-Value Cache Compression with Predictive Online Pruning
arXiv:2607. 05061v1 Announce Type: new Abstract: Key-value (KV) cache growth is a major bottleneck in autoregressive decoding, as memory and bandwidth scale linearly with context length.
arXiv Machine Learning
Jun 2
ForesightKV: Optimizing KV Cache Eviction for Reasoning Models by Learning Long-Term Contribution
arXiv:2602. 03203v2 Announce Type: replace-cross Abstract: Recently, large language models (LLMs) have shown remarkable reasoning abilities by producing long reasoning traces.
arXiv Machine Learning
Jun 3
Value-Aware Stochastic KV Cache Eviction for Reasoning Models
arXiv:2606. 03928v1 Announce Type: new Abstract: Reasoning models improve accuracy through extended chains of thought, but their long outputs create a memory and compute bottleneck.