arXiv Computation and Language
Fast KVzip: Efficient and Accurate LLM Inference with Gated KV Eviction
Read the original on arXiv Computation and Language →The Flow has not summarised this story yet — read it at arXiv Computation and Language.
Related stories
arXiv AI
Jul 3
Kara: Efficient Reasoning LLM Serving via Sliding-Window KV Cache Compression
arXiv:2607. 01237v1 Announce Type: cross Abstract: Reasoning language models often generate long chain-of-thought (CoT), which accumulates a massive KV cache during the decoding phase and incurs high decoding latency and limited throughput.
arXiv AI
Jul 7
KARA: Efficient Reasoning LLM Serving via Sliding-Window KV Cache Compression
arXiv:2607. 01237v2 Announce Type: replace-cross Abstract: Reasoning language models often generate long chain-of-thought (CoT), which accumulates a massive KV cache during the decoding phase and incurs high decoding latency and limited throughput.
arXiv Machine Learning
Jun 29
Learning to Evict from Key-Value Cache
arXiv:2602. 10238v2 Announce Type: replace-cross Abstract: The growing size of Large Language Models (LLMs) makes efficient inference challenging, primarily due to the memory demands of the autoregressive Key-Value (KV) cache.
arXiv Computation and Language
Sep 4
SGD-KV: Summarization Guided KV Cache Compression
SGD-KV is a head‑aware framework for compressing key‑value caches in large language models. It uses a chunk‑summarization diagnostic task to identify attention heads that specialize in hierarchical information aggregation, allowing the KV cache budget to be allocated based on each head’s summarization score. Experiments on Qwen2.5‑7B‑1M and Qwen3‑32B show state‑of‑the‑art performance on up to 1M‑token contexts while cutting KV cache memory usage by up to 75%.
Hugging Face Trending Papers
Sep 3
SGD-KV: Summarization Guided KV Cache Compression
SGD-KV is a head‑aware framework that compresses key‑value caches in large language models by using a chunk‑summarization diagnostic task to identify attention heads that specialize in hierarchical information aggregation. It prioritizes these heads during compression, achieving state‑of‑the‑art performance on long‑context benchmarks with up to 1M tokens while cutting KV cache memory usage by as much as 75%. Experiments on Qwen2.5‑7B‑1M and Qwen3‑32B confirm that allocating cache budget based on summarization scores yields a superior efficiency‑accuracy trade‑off for long‑context inference.
arXiv AI
Jun 30
Coverage-Driven KV Cache Eviction for Efficient and Improved Inference of LLM
arXiv:2606. 29563v1 Announce Type: cross Abstract: Large language models (LLMs) excel at complex tasks like question answering and summarization, thanks to their ability to handle long-context inputs.