The paper addresses the challenge of long input contexts in Retrieval-Augmented Generation (RAG) systems, where concatenating many retrieved chunks increases prefill workload and time to first token (TTFT). It proposes a dual strategy: fine‑tuning the model to be aware of KV cache concatenation and selectively recomputing only part of the KV caches. Experiments on the RULER benchmark show that for a 124k‑token input, this combined method boosts the RULER score by 9.7 points over a baseline that recomputes caches only, while cutting TTFT by 80% compared with full attention.
By Fumihiko Tachibana, Daisuke Miyashita, Jun Deguchi
arXiv:2607. 27600v1 Announce Type: new Abstract: Key-value (KV) cache management through compression and eviction strategies has emerged as an important research direction in recent years.
By Stephen Gould, Anton van den Hengel
arXiv:2608.21362v1 Announce Type: new
Abstract: Transformer-based large language models (LLMs) incur high prefill latency because key-value (KV) tensors must be recomputed for each request. Existing...
By Srihari Unnikrishnan
arXiv:2607. 28069v2 Announce Type: replace Abstract: Long-context retrieval and agentic workloads repeatedly reuse the same documents under changing instructions, histories, and document orders.
By Hui Xie, Peng Xiao, Yutong Deng, Shuoran Dou, Jian Yang, Jinyang Guo
arXiv:2606. 17872v1 Announce Type: cross Abstract: Large language models (LLMs) outperform earlier architectures on generative inference and long-context tasks, but their large size introduces significant challenges in memory usage, energy cost, and on-device deployment.
By Ning Ni, Yingjie Lao
arXiv:2512.17452v4 Announce Type: replace-cross
Abstract: Long-context LLM inference is bottlenecked by the quadratic attention complexity and linear Key-Value (KV) cache growth. Prior approaches mit...
By Yen-Chieh Huang, Pi-Cheng Hsiu, Rui Fang, Ming-Syan Chen
KV-Repurpose is an open‑source, vLLM‑based system that enables segment‑wise KV‑cache reuse beyond exact prompt prefixes in large language model serving. It integrates cache‑aware segment scheduling, prompt construction, capacity management, workload generation, and runtime monitoring to evaluate non‑prefix KV‑cache reuse. Experiments show that naive segment reuse maintains output quality for single‑document QA and summarization, while significantly improving cache hit rates and reducing time‑to‑first‑token compared to prefix caching, especially under Zipfian RAG‑style workloads.
By Taiga Ikeda, Daisuke Miyashita, Jun Deguchi
arXiv:2608.30295v1 Announce Type: cross
Abstract: Large language models (LLMs) have demonstrated strong capabilities in handling long-context tasks, but processing such long contexts remains challeng...
By Haoyun Jiang, Haolin Li, Jianwei Zhang, Fei Huang, Qiang Hu, Minmin Sun, Shuai Xiao, Yong Li, Junyang Lin, Jiangchao Yao
arXiv:2601.17668v3 Announce Type: replace-cross
Abstract: Efficient key-value (KV) cache management is crucial for the practical deployment of large language models (LLMs), yet existing compression t...
By Jang-Hyun Kim, Dongyoon Han, Sangdoo Yun
arXiv:2606. 06256v1 Announce Type: new Abstract: As the input length of large language model (LLM) serving continues to grow, the KV cache has become a dominant bottleneck in AI infrastructure.
By Yang Liu, ZhaoKai Luo, HuaYi Jin, ZhiYong Wang, RuoZhou He, BoYu Wang, Guanjie Chen, Junhao Hu
DropKV introduces a new KV‑cache eviction framework that decouples residual‑output perturbation into independent per‑token scores, avoiding the NP‑hard combinatorial optimization of prior methods. The approach offers a provable constant‑factor approximation under a cone condition verified on three long‑context LLMs, and a high‑performance Triton‑based implementation that yields a 19.8× speedup in scoring kernels and up to 9.9% end‑to‑end prefill speedup. Benchmarks on RULER and LongBench show DropKV consistently outperforms state‑of‑the‑art baselines at the same cache budgets, achieving the lowest inference latency among eviction strategies while remaining faster than dynamic cache baselines.
By Aozhong Zhang, Selcuk Gurses, Yanxia Deng, Naigang Wang, Chi-Chun Liu, Davis Wertheimer, Derrick Liu, Xin Li, Zi Yang, Felix X. -F. Ye, Penghang Yin
arXiv:2604.05012v2 Announce Type: replace-cross
Abstract: Efficient inference with Large Language Models (LLMs) increasingly relies on Key-Value (KV) caches to store previously computed key and value...
By Oteo Mamo, Olga Kogiou, Hyunjin Yi, Weikuan Yu