The paper addresses the challenge of long input contexts in Retrieval-Augmented Generation (RAG) systems, where concatenating many retrieved chunks increases prefill workload and time to first token (TTFT). It proposes a dual strategy: fine‑tuning the model to be aware of KV cache concatenation and selectively recomputing only part of the KV caches. Experiments on the RULER benchmark show that for a 124k‑token input, this combined method boosts the RULER score by 9.7 points over a baseline that recomputes caches only, while cutting TTFT by 80% compared with full attention.
arXiv:2604.13226v3 Announce Type: replace-cross Abstract: Large Language Models (LLMs) rely heavily on Key-Value (KV) caching to minimize inference latency. However, standard KV caches are context-de...
arXiv:2607. 06519v1 Announce Type: new Abstract: Long-context LLM inference is increasingly limited by the memory and bandwidth cost of KV caches, yet aggressive compression can remove the layer-specific evidence needed for retrieval and multi-step reasoning.
KV-Repurpose is an open‑source, vLLM‑based system that enables segment‑wise KV‑cache reuse beyond exact prompt prefixes in large language model serving. It integrates cache‑aware segment scheduling, prompt construction, capacity management, workload generation, and runtime monitoring to evaluate non‑prefix KV‑cache reuse. Experiments show that naive segment reuse maintains output quality for single‑document QA and summarization, while significantly improving cache hit rates and reducing time‑to‑first‑token compared to prefix caching, especially under Zipfian RAG‑style workloads.
KV$^2$ is a query‑agnostic key‑value cache compression technique that selectively reconstructs only informative in‑context tokens using a lightweight proxy scorer before final eviction scoring. On benchmarks such as RULER, Needle‑in‑a‑Haystack, and LongBench, KV$^2$ outperforms baseline methods, especially under tight memory budgets, achieving higher scores with lower runtime and peak memory than full‑context reconstruction. The approach demonstrates that reusable KV‑cache compression can avoid reprocessing the entire prompt while maintaining quality.
RIFT introduces selection‑based KV cache compression strategies that balance prefill information with recently generated tokens to mitigate decode staleness in speculative decoding. By refreshing the KV cache selection during token generation and employing an adaptive refresh strategy, RIFT improves drafter quality while managing decode latency. A compressibility‑driven drafter training phase further enhances performance, enabling Qwen3.5‑4B to retain over 98% of its full‑cache acceptance length within an 8K‑token KV cache budget on long‑context inputs.
arXiv:2609.39329v1 Announce Type: new Abstract: Long-context inference with Large Language Models (LLMs) is bottlenecked by the linearly growing memory of the key-value (KV) cache. Existing compressi...
KV-Repurpose is an open‑source, vLLM‑based system that enables segment‑wise KV‑cache reuse beyond exact prompt prefixes in large language model serving. It integrates cache‑aware segment scheduling, prompt construction, capacity management, workload generation, and runtime monitoring. Experiments show that while naive segment reuse maintains output quality for single‑document tasks, it reveals limitations in cross‑document reasoning, and under a Zipfian RAG‑style workload it achieves a 63.9% token‑level cache hit rate versus 3.3% for prefix caching at 30 GB, dramatically lowering time‑to‑first‑token.
The paper investigates KV cache offloading for long‑context large language models, focusing on tasks that require extensive information extraction from the prompt. The authors introduce the Text2JSON benchmark, a highly context‑intensive task that demands structured knowledge extraction from raw text, and evaluate modern KV offloading techniques on this benchmark and other similar tasks. Their experiments on Llama 3 and Qwen 3 reveal significant accuracy degradation, attributing it to low‑rank key projection and unreliable landmarks, and propose a simpler strategy that markedly improves performance across multiple LLM families.
arXiv:2605.23258v2 Announce Type: replace Abstract: KV cache growth is a major bottleneck for long-context inference in large language models. Existing methods are often dominated by binary eviction...
arXiv:2607. 27600v1 Announce Type: new Abstract: Key-value (KV) cache management through compression and eviction strategies has emerged as an important research direction in recent years.
arXiv:2608.21362v1 Announce Type: new Abstract: Transformer-based large language models (LLMs) incur high prefill latency because key-value (KV) tensors must be recomputed for each request. Existing...
arXiv:2609.07966v1 Announce Type: new Abstract: Key--value (KV) cache compression is an effective way to reduce the memory overhead of large language model (LLM) inference, particularly for long-cont...