arXiv AI
KVBoost: Chunk-Level Key-Value Cache Reuse with Deviation-Guided Recomputation for Efficient Large Language Model Inference
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Aug 26
PuzzleKV: Page-Wise Low-Rank Decomposition for KV Cache Compression
arXiv:2608.23843v1 Announce Type: new Abstract: Long-context inference in large language models (LLMs) is increasingly limited by the memory required for the key-value (KV) cache. KV cache compressio...
arXiv Machine Learning
Oct 8
KVFetch: Temporal Prefetching for the Missing Half of KV Cache Compression
arXiv:2610.08811v1 Announce Type: new Abstract: As context windows scale to tens or hundreds of thousands of tokens, KV cache compression has become essential for efficient LLM inference. Existing me...
arXiv Machine Learning
2d ago
A Simple Plug-in for Improving Eviction-Based KV Cache Compression
arXiv:2605.23258v2 Announce Type: replace Abstract: KV cache growth is a major bottleneck for long-context inference in large language models. Existing methods are often dominated by binary eviction...
Hugging Face Trending Papers
Oct 5
DeferKV: Rethinking Eviction Timing for One-Shot KV Cache Compression
DeferKV rethinks when to evict key‑value (KV) cache entries in long‑context large language models. By delaying eviction until the first decoding step and combining prompt‑side and decode‑side attention signals, it aligns KV importance estimation with actual generation needs. The method requires no extra training or modules and consistently improves performance on benchmarks while keeping latency low.
arXiv AI
Jul 3
Kara: Efficient Reasoning LLM Serving via Sliding-Window KV Cache Compression
arXiv:2607. 01237v1 Announce Type: cross Abstract: Reasoning language models often generate long chain-of-thought (CoT), which accumulates a massive KV cache during the decoding phase and incurs high decoding latency and limited throughput.
arXiv AI
Jun 9
Enabling KV Caching of Shared Prefix for Diffusion Language Models
arXiv:2606. 07571v1 Announce Type: cross Abstract: Key-value (KV) caching for shared prefixes is essential for high-throughput large language model (LLM) serving, but it faces critical challenges in emerging diffusion language models (DLMs).