arXiv:2607. 24331v1 Announce Type: new Abstract: As the inference phase of Large Language Models (LLMs) requires handling long context windows, the Key-Value (KV) cache initially appears to address this challenge but eventually becomes a significant bottleneck as the context window continues to grow.
arXiv:2607. 16213v1 Announce Type: new Abstract: Large Language Models (LLMs) generate text autoregressively, relying on a key-value (KV) cache whose memory footprint grows linearly with context length, creating a major bottleneck.
arXiv:2608.23843v1 Announce Type: new Abstract: Long-context inference in large language models (LLMs) is increasingly limited by the memory required for the key-value (KV) cache. KV cache compressio...
SGD-KV is a head‑aware framework for compressing key‑value caches in large language models. It uses a chunk‑summarization diagnostic task to identify attention heads that specialize in hierarchical information aggregation, allowing the KV cache budget to be allocated based on each head’s summarization score. Experiments on Qwen2.5‑7B‑1M and Qwen3‑32B show state‑of‑the‑art performance on up to 1M‑token contexts while cutting KV cache memory usage by up to 75%.
SGD-KV is a head‑aware framework that compresses key‑value caches in large language models by using a chunk‑summarization diagnostic task to identify attention heads that specialize in hierarchical information aggregation. It prioritizes these heads during compression, achieving state‑of‑the‑art performance on long‑context benchmarks with up to 1M tokens while cutting KV cache memory usage by as much as 75%. Experiments on Qwen2.5‑7B‑1M and Qwen3‑32B confirm that allocating cache budget based on summarization scores yields a superior efficiency‑accuracy trade‑off for long‑context inference.
arXiv:2608. 02691v1 Announce Type: cross Abstract: The key-value (KV) cache has become a major memory and bandwidth bottleneck in long-context large language model inference, making ultra-low-bit quantization increasingly important.
arXiv:2609.13285v1 Announce Type: cross Abstract: The KV cache is a primary bottleneck for Transformer decoding: its memory footprint and cache-read traffic grow with sequence length. Grouped-query a...
arXiv:2609.36835v1 Announce Type: new Abstract: Long-context large language model inference is bottlenecked by KV caches that grow linearly with sequence length. This burden is especially severe for...
arXiv:2604.11501v2 Announce Type: replace-cross Abstract: Rank reduction discards dimensions; quantization keeps them at lower precision. Comparing the two requires a choice of what compression shoul...
arXiv:2606.14782v3 Announce Type: replace-cross Abstract: Multimodal Large Language Models (MLLMs) achieve strong vision-language reasoning but incur large KV caches and high decoding latency with lo...
Long-context large language model (LLM) inference is increasingly constrained by the memory footprint and decoding cost of key-value (KV) caches, limiting sustainable deployment on resource-constrained hardware. Existing KV cache eviction methods typically apply heuristic token scoring over all heads in GQA-based LLMs.
arXiv:2606. 29563v1 Announce Type: cross Abstract: Large language models (LLMs) excel at complex tasks like question answering and summarization, thanks to their ability to handle long-context inputs.
arXiv:2606. 24467v1 Announce Type: new Abstract: Long-context large language model (LLM) inference is increasingly constrained by the memory footprint and decoding cost of key-value (KV) caches, limiting sustainable deployment on resource-constrained hardware.