arXiv Machine Learning
Jul 3
The risk of KV cache compression
arXiv:2607. 01520v1 Announce Type: new Abstract: Transformer inference on long sequences is expensive because softmax attention repeatedly reads from a large KV cache.
ReFreeKV: Towards Threshold-Free KV Cache Compression
arXiv:2502. 16886v4 Announce Type: replace-cross Abstract: To reduce memory consumption during LLM inference, a handful of methods have been proposed for KV cache pruning.
Related stories
arXiv AI
Jun 30
HARD-KV: Head-Adaptive Regularization for Decoding-time KV Compression
arXiv:2606. 28831v1 Announce Type: cross Abstract: Long-context LLM inference faces a fundamental conflict: head-adaptive compression algorithms (e.
arXiv AI
Jul 3
Kara: Efficient Reasoning LLM Serving via Sliding-Window KV Cache Compression
arXiv:2607. 01237v1 Announce Type: cross Abstract: Reasoning language models often generate long chain-of-thought (CoT), which accumulates a massive KV cache during the decoding phase and incurs high decoding latency and limited throughput.
arXiv Machine Learning
Jul 27
HiKV: Hierarchical Importance-Aware KV Cache with Hardware Acceleration for LLM Decoding
arXiv:2607. 22389v1 Announce Type: cross Abstract: With the rapid adoption of long-context large language models (LLMs), the continuously growing KV cache during decoding has become the critical memory bottleneck.
arXiv Machine Learning
Jul 31
Back from the Future: Key-Value Cache Management by Counter-Causal Surprise
arXiv:2607. 27600v1 Announce Type: new Abstract: Key-value (KV) cache management through compression and eviction strategies has emerged as an important research direction in recent years.
arXiv AI
Jul 8
FreqDepthKV: Frequency-Guided Depth Sharing for Robust KV Cache Compression in Long-Context LLM Inference
arXiv:2607. 06519v1 Announce Type: new Abstract: Long-context LLM inference is increasingly limited by the memory and bandwidth cost of KV caches, yet aggressive compression can remove the layer-specific evidence needed for retrieval and multi-step reasoning.
arXiv AI
Jul 7
KARA: Efficient Reasoning LLM Serving via Sliding-Window KV Cache Compression
arXiv:2607. 01237v2 Announce Type: replace-cross Abstract: Reasoning language models often generate long chain-of-thought (CoT), which accumulates a massive KV cache during the decoding phase and incurs high decoding latency and limited throughput.
arXiv AI
Jun 16
PolyKV: Heterogeneous Retention and Allocation for KV Cache Compression
arXiv:2606. 15157v1 Announce Type: cross Abstract: KV cache compression is essential for reducing the memory cost of long-context large language model inference.
arXiv AI
Aug 11
RippleKV: Cross-Layer KV Cache Allocation via Perturbation Propagation
arXiv:2608. 08684v1 Announce Type: cross Abstract: Long-context LLM inference is bottlenecked by KV cache memory, yet distributing a limited cache budget across layers remains challenging.
arXiv Machine Learning
Jun 3
Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Serving
arXiv:2606. 02964v1 Announce Type: cross Abstract: Large Language Model (LLM) inference relies on key-value (KV) caches to avoid redundant attention computation.
arXiv AI
Jun 9
End-to-End Context Compression at Scale
arXiv:2606. 09659v1 Announce Type: cross Abstract: Long-context language model inference is bottlenecked by memory, as the KV cache grows with context length.
arXiv AI
Jun 9
FlashMemory-DeepSeek-V4: Lightning Index Ultra-Long Context via Lookahead Sparse Attention
arXiv:2606. 09079v1 Announce Type: cross Abstract: Conventional LLMs keep the full KV cache loaded during decoding, causing a severe GPU memory bottleneck for ultra-long context serving.