← Back to all news
arXiv Machine Learning By Samuel Cestola, Tianxiang Xia, Pengfei Zheng, Weiyan Zheng, Bo Wang, Yi Zhao, Diego Didona

Evaluating the accuracy of KV cache reuse techniques

Read the original on arXiv Machine Learning →

The paper discusses position‑independent KV cache reuse, a technique designed to cut latency in retrieval‑augmented generation by reusing chunk‑level KV caches across prompts. It argues that current evaluation methods overstate the accuracy of such reuse because they do not accurately capture the loss of accuracy, and that existing datasets lack the necessary reuse dynamics for thorough testing. To remedy this, the authors propose a new evaluation methodology that unambiguously measures accuracy loss and introduce Boxoffice, a tool that programmatically creates datasets with challenging KV cache reuse patterns.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Computation and Language
Sep 10

Fine-Tuning a KV Cache Concatenation-Aware Model or Recomputing KV Caches? Why Not Both?

The paper addresses the challenge of long input contexts in Retrieval-Augmented Generation (RAG) systems, where concatenating many retrieved chunks increases prefill workload and time to first token (TTFT). It proposes a dual strategy: fine‑tuning the model to be aware of KV cache concatenation and selectively recomputing only part of the KV caches. Experiments on the RULER benchmark show that for a 124k‑token input, this combined method boosts the RULER score by 9.7 points over a baseline that recomputes caches only, while cutting TTFT by 80% compared with full attention.

By Fumihiko Tachibana, Daisuke Miyashita, Jun Deguchi
ragfine-tuningefficiencybenchmarks
More like this →
arXiv Machine Learning
5d ago

CacheReforge: Bounded Recovery for Stale KV Caches under Evolving Adapters

CacheReforge is a method for recovering stale key‑value (KV) caches in large language models when lightweight adapters evolve. It represents stale caches as layer‑wise mixed‑version objects and uses adapter anchors, sensitivity calibration, drift accumulation, and restart boundaries to decide between direct reuse, bounded recomputation, or full suffix recovery. Experiments on Qwen2.5 models with continual LoRA updates show a 92.4% reduction in mean KL divergence while only recomputing 5.44% of layers and cutting cache‑maintenance time by 93.2% compared to full prefill.

By Yuhang Cao, Yanzhou Mu, Chunrong Fang, Zhenyu Chen
llmsfine-tuning
More like this →
arXiv AI
Jul 8

Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context Serving

arXiv:2607. 05399v1 Announce Type: cross Abstract: Large language model serving is increasingly limited by KV-cache growth under long-context workloads, yet existing KV-cache compression techniques are difficult to compare because they were evaluated on different models, tasks, budgets, and serving stacks.

By Nikita Agrawal, Ruben Mayer
llmsnlpefficiencybenchmarks
More like this →
arXiv Machine Learning
Sep 18

PrefixBench-H100: Characterizing Prefix Reuse and Time-to-First-Token in H100 LLM Serving

PrefixBench-H100 is a reproducible benchmark that evaluates how reusing prompt prefixes affects LLM serving performance on NVIDIA H100 GPUs. It tests two popular runtimes (vLLM and TensorRT-LLM) across varied workloads, measuring metrics such as time-to-first-token, latency, throughput, cache hits, and GPU memory usage. The study identifies when prefix reuse significantly reduces first‑token latency and when cache pressure diminishes those gains, noting that cache effectiveness is largely unaffected by concurrency or output length, while differences arise mainly in scheduling.

By Omkar Shewale, Deepak Kumar, Divakar Kumar Yadav
llmsragagentsefficiencybenchmarks
More like this →