arXiv:2604.13226v3 Announce Type: replace-cross
Abstract: Large Language Models (LLMs) rely heavily on Key-Value (KV) caching to minimize inference latency. However, standard KV caches are context-de...
By Chuangtao Chen, Grace Li Zhang, Xunzhao Yin, Cheng Zhuo, Bing Li, Ulf Schlichtmann
arXiv:2607. 06519v1 Announce Type: new Abstract: Long-context LLM inference is increasingly limited by the memory and bandwidth cost of KV caches, yet aggressive compression can remove the layer-specific evidence needed for retrieval and multi-step reasoning.
By Anna C\'ordoba, Adam Puente Tercero, Nerea Angulo Hijo, Mar Linares Tercero, Julia Barrientos, Ainhoa Miranda, Jes\'us Olivera
KV-Repurpose is an open‑source, vLLM‑based system that enables segment‑wise KV‑cache reuse beyond exact prompt prefixes in large language model serving. It integrates cache‑aware segment scheduling, prompt construction, capacity management, workload generation, and runtime monitoring to evaluate non‑prefix KV‑cache reuse. Experiments show that naive segment reuse maintains output quality for single‑document QA and summarization, while significantly improving cache hit rates and reducing time‑to‑first‑token compared to prefix caching, especially under Zipfian RAG‑style workloads.
By Taiga Ikeda, Daisuke Miyashita, Jun Deguchi
KV$^2$ is a query‑agnostic key‑value cache compression technique that selectively reconstructs only informative in‑context tokens using a lightweight proxy scorer before final eviction scoring. On benchmarks such as RULER, Needle‑in‑a‑Haystack, and LongBench, KV$^2$ outperforms baseline methods, especially under tight memory budgets, achieving higher scores with lower runtime and peak memory than full‑context reconstruction. The approach demonstrates that reusable KV‑cache compression can avoid reprocessing the entire prompt while maintaining quality.
By Johannes Wesch, Danni Liu, Jan Niehues
RIFT introduces selection‑based KV cache compression strategies that balance prefill information with recently generated tokens to mitigate decode staleness in speculative decoding. By refreshing the KV cache selection during token generation and employing an adaptive refresh strategy, RIFT improves drafter quality while managing decode latency. A compressibility‑driven drafter training phase further enhances performance, enabling Qwen3.5‑4B to retain over 98% of its full‑cache acceptance length within an 8K‑token KV cache budget on long‑context inputs.
By Dor Tsur, Maor Ashkenazi, Maximilian Golub, Peter Hicks, Yaniv Galron, Talor Abramovich, Yoav Miron, Tiyasa Mitra, Bita Darvish Rouhani, Ran Zilberstein
arXiv:2609.39329v1 Announce Type: new
Abstract: Long-context inference with Large Language Models (LLMs) is bottlenecked by the linearly growing memory of the key-value (KV) cache. Existing compressi...
By Chanryeol Lee, Chanhyuk Lee, Yeonwoo Choi, Donggyun Kim, Seunghoon Hong