← Back to all news
arXiv Computation and Language By Fumihiko Tachibana, Daisuke Miyashita, Jun Deguchi

Fine-Tuning a KV Cache Concatenation-Aware Model or Recomputing KV Caches? Why Not Both?

Read the original on arXiv Computation and Language →

The paper addresses the challenge of long input contexts in Retrieval-Augmented Generation (RAG) systems, where concatenating many retrieved chunks increases prefill workload and time to first token (TTFT). It proposes a dual strategy: fine‑tuning the model to be aware of KV cache concatenation and selectively recomputing only part of the KV caches. Experiments on the RULER benchmark show that for a 124k‑token input, this combined method boosts the RULER score by 9.7 points over a baseline that recomputes caches only, while cutting TTFT by 80% compared with full attention.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv AI
Jul 8

FreqDepthKV: Frequency-Guided Depth Sharing for Robust KV Cache Compression in Long-Context LLM Inference

arXiv:2607. 06519v1 Announce Type: new Abstract: Long-context LLM inference is increasingly limited by the memory and bandwidth cost of KV caches, yet aggressive compression can remove the layer-specific evidence needed for retrieval and multi-step reasoning.

By Anna C\'ordoba, Adam Puente Tercero, Nerea Angulo Hijo, Mar Linares Tercero, Julia Barrientos, Ainhoa Miranda, Jes\'us Olivera
llmsnlpefficiencybenchmarks
More like this →
arXiv AI
Sep 2

KV Cache Offloading for Context-Intensive Tasks

The paper investigates KV cache offloading for long‑context large language models, focusing on tasks that require extensive information extraction from the prompt. The authors introduce the Text2JSON benchmark, a highly context‑intensive task that demands structured knowledge extraction from raw text, and evaluate modern KV offloading techniques on this benchmark and other similar tasks. Their experiments on Llama 3 and Qwen 3 reveal significant accuracy degradation, attributing it to low‑rank key projection and unreliable landmarks, and propose a simpler strategy that markedly improves performance across multiple LLM families.

By Andrey Bocharnikov, Ivan Ermakov, Denis Kuznedelev, Vyacheslav Zhdanovskiy, Yegor Yershov
llmsefficiencybenchmarks
More like this →
arXiv AI
Jun 24

CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference

arXiv:2606. 24467v1 Announce Type: new Abstract: Long-context large language model (LLM) inference is increasingly constrained by the memory footprint and decoding cost of key-value (KV) caches, limiting sustainable deployment on resource-constrained hardware.

By Xiaolin Lin, Jingcun Wang, Olga Kondrateva, Yiyu Shi, Bing Li, Grace Li Zhang
llmsefficiency
More like this →