arXiv Machine Learning
WakeKV: Reactive, Reversible KV Residency for Heads That Change Their Minds
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Computation and Language
Sep 4
Random Attention: Rethinking KV Cache Eviction for Efficient Reasoning
The paper introduces Random Attention, a method that evicts KV cache entries uniformly at random within each attention head while preserving the prompt. Experiments on four models and six reasoning tasks show that this simple strategy matches the performance of the best existing eviction methods and achieves 32‑43% higher throughput in vLLM deployments. The authors explain that the prompt is the most fragile cache component and that reasoning traces are redundantly stored across text and attention heads, making a selection score unnecessary.
arXiv AI
Jun 9
From Rigid to Dynamic: Entropy-Guided Adaptive Inference for Long-Context LLMs
arXiv:2606. 09508v1 Announce Type: new Abstract: Existing sparse attention and KV cache compression methods for long-context LLM inference typically apply fixed sparsity patterns or uniform budgets across all attention heads, overlooking the substantial variation in attention behavior among heads and contexts.
arXiv Machine Learning
5d ago
When to Evict, Not What to Keep: Draft-Guided Eviction for Training-Free KV-Cache Compression
The paper introduces Draft‑Guided Eviction (DGE), a training‑free KV‑cache compression technique that postpones token eviction until after drafting the first two answer tokens, thereby preserving the cache entries needed for the answer trajectory. Unlike prior methods that focus on optimizing what to keep, DGE changes the timing of eviction, allowing existing compression schemes such as SnapKV, H2O, and PyramidKV to be applied without altering their eviction scores. Experiments show DGE consistently outperforms previous approaches across multiple instruct‑tuned models, achieving near‑full‑cache performance on LongBench and demonstrating that the improvement stems from trajectory anchoring rather than changes in cache selection.
arXiv AI
Sep 3
HeadWiseKV: Budgeted Per-Head Cache Residency for Hybrid Long-Context Language Models
HeadWiseKV is a training‑free framework that compresses the residual global key–value caches of hybrid long‑context language models by assigning each physical KV head a static, multilevel history window. It formulates cache allocation as a restricted operational rate–distortion problem and uses the SeqCalib algorithm to generate per‑head residency policies that account for interactions across layers. In evaluations on four hybrid models, HeadWiseKV preserves near‑full‑KV quality while reducing peak device memory usage by 8.59% at a 112K context length and extending the largest verified context from 114K to 161K.
Hugging Face Trending Papers
Jun 23
CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference
Long-context large language model (LLM) inference is increasingly constrained by the memory footprint and decoding cost of key-value (KV) caches, limiting sustainable deployment on resource-constrained hardware. Existing KV cache eviction methods typically apply heuristic token scoring over all heads in GQA-based LLMs.
arXiv AI
Jun 6
RedKnot: Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttention
arXiv:2606. 06256v1 Announce Type: new Abstract: As the input length of large language model (LLM) serving continues to grow, the KV cache has become a dominant bottleneck in AI infrastructure.