arXiv:2607. 05399v1 Announce Type: cross Abstract: Large language model serving is increasingly limited by KV-cache growth under long-context workloads, yet existing KV-cache compression techniques are difficult to compare because they were evaluated on different models, tasks, budgets, and serving stacks.
arXiv:2608.23843v1 Announce Type: new Abstract: Long-context inference in large language models (LLMs) is increasingly limited by the memory required for the key-value (KV) cache. KV cache compressio...
SGD-KV is a head‑aware framework for compressing key‑value caches in large language models. It uses a chunk‑summarization diagnostic task to identify attention heads that specialize in hierarchical information aggregation, allowing the KV cache budget to be allocated based on each head’s summarization score. Experiments on Qwen2.5‑7B‑1M and Qwen3‑32B show state‑of‑the‑art performance on up to 1M‑token contexts while cutting KV cache memory usage by up to 75%.
The paper investigates KV cache offloading for long‑context large language models, focusing on tasks that require extensive information extraction from the prompt. The authors introduce the Text2JSON benchmark, a highly context‑intensive task that demands structured knowledge extraction from raw text, and evaluate modern KV offloading techniques on this benchmark and other similar tasks. Their experiments on Llama 3 and Qwen 3 reveal significant accuracy degradation, attributing it to low‑rank key projection and unreliable landmarks, and propose a simpler strategy that markedly improves performance across multiple LLM families.
arXiv:2609.39329v1 Announce Type: new Abstract: Long-context inference with Large Language Models (LLMs) is bottlenecked by the linearly growing memory of the key-value (KV) cache. Existing compressi...
arXiv:2607. 16213v1 Announce Type: new Abstract: Large Language Models (LLMs) generate text autoregressively, relying on a key-value (KV) cache whose memory footprint grows linearly with context length, creating a major bottleneck.
arXiv:2609.37988v1 Announce Type: new Abstract: As the context size of text processed with an LLM grows, the size of KV caches can outstrip the memory allocated for the original model weights. This i...
SGD-KV is a head‑aware framework that compresses key‑value caches in large language models by using a chunk‑summarization diagnostic task to identify attention heads that specialize in hierarchical information aggregation. It prioritizes these heads during compression, achieving state‑of‑the‑art performance on long‑context benchmarks with up to 1M tokens while cutting KV cache memory usage by as much as 75%. Experiments on Qwen2.5‑7B‑1M and Qwen3‑32B confirm that allocating cache budget based on summarization scores yields a superior efficiency‑accuracy trade‑off for long‑context inference.
arXiv:2607. 06519v1 Announce Type: new Abstract: Long-context LLM inference is increasingly limited by the memory and bandwidth cost of KV caches, yet aggressive compression can remove the layer-specific evidence needed for retrieval and multi-step reasoning.
Long-context large language model (LLM) inference is increasingly constrained by the memory footprint and decoding cost of key-value (KV) caches, limiting sustainable deployment on resource-constrained hardware. Existing KV cache eviction methods typically apply heuristic token scoring over all heads in GQA-based LLMs.
arXiv:2606. 24467v1 Announce Type: new Abstract: Long-context large language model (LLM) inference is increasingly constrained by the memory footprint and decoding cost of key-value (KV) caches, limiting sustainable deployment on resource-constrained hardware.
arXiv:2609.07966v1 Announce Type: new Abstract: Key--value (KV) cache compression is an effective way to reduce the memory overhead of large language model (LLM) inference, particularly for long-cont...
The paper addresses the challenge of long input contexts in Retrieval-Augmented Generation (RAG) systems, where concatenating many retrieved chunks increases prefill workload and time to first token (TTFT). It proposes a dual strategy: fine‑tuning the model to be aware of KV cache concatenation and selectively recomputing only part of the KV caches. Experiments on the RULER benchmark show that for a 124k‑token input, this combined method boosts the RULER score by 9.7 points over a baseline that recomputes caches only, while cutting TTFT by 80% compared with full attention.