CacheBridge is a method for efficiently transferring key‑value (KV) caches between large language models (LLMs) in a multi‑model system. It replaces the full‑head mapping approach by matching each target KV head to a single source head, weighting reconstruction errors by causal attention sensitivity, and building weighted sufficient statistics with a fused GPU kernel. The technique achieves comparable or better accuracy to full‑head mapping while reducing mapper storage by up to eight‑fold, accelerating application by up to three‑times, and cutting construction time dramatically.
By Xingyu Qu, Siyuan Lu, Zhiyu Chen, Sheng Wang, Tao Lin
arXiv:2609.32259v2 Announce Type: replace
Abstract: Recent multi-agent LLM systems increasingly combine heterogeneous models for specialized agent roles. However, text-based communication requires ea...
By Vincent-Daniel Yun, Woosang Lim, Haneul Yoo, Sungjoo Yoo, Murali Annavaram, Sai Praneeth Karimireddy
arXiv:2608.30963v1 Announce Type: cross
Abstract: Modern large language model (LLM) serving systems increasingly operate over repeated or shared context, yet each model typically performs its own pre...
By Yi Li, Dongming Jiang, Yi Zhao, Bingzhe Li
arXiv:2608. 08684v1 Announce Type: cross Abstract: Long-context LLM inference is bottlenecked by KV cache memory, yet distributing a limited cache budget across layers remains challenging.
By Dongjie Xu, Kai Qian, Julius, Weijie Shi, Yuxuan Sun, Minghua Tang, Fenglei Jin, Hanchi Dong, Jiajie Xu
arXiv:2609.17109v1 Announce Type: new
Abstract: A common small-model deployment runs one shared backbone with several LoRA specialists that answer over the same context. Serving them naively re-prefi...
By Dushyant Rajput
arXiv:2609.39329v1 Announce Type: new
Abstract: Long-context inference with Large Language Models (LLMs) is bottlenecked by the linearly growing memory of the key-value (KV) cache. Existing compressi...
By Chanryeol Lee, Chanhyuk Lee, Yeonwoo Choi, Donggyun Kim, Seunghoon Hong
arXiv:2606. 15157v1 Announce Type: cross Abstract: KV cache compression is essential for reducing the memory cost of long-context large language model inference.
By Chao Fei, Panos Kalnis
arXiv:2608.21362v1 Announce Type: new
Abstract: Transformer-based large language models (LLMs) incur high prefill latency because key-value (KV) tensors must be recomputed for each request. Existing...
By Srihari Unnikrishnan
A common small-model deployment runs one shared backbone with several LoRA specialists that answer over the same context. Serving them naively re-prefills that shared context once per specialist. We s...
arXiv:2609.10266v1 Announce Type: new
Abstract: LLM serving systems already reuse KV caches, but only when the reused text sits at the very start of the prompt. Two growing workloads break this condi...
By Xi Shi, Qian Lou
The paper addresses the challenge of long input contexts in Retrieval-Augmented Generation (RAG) systems, where concatenating many retrieved chunks increases prefill workload and time to first token (TTFT). It proposes a dual strategy: fine‑tuning the model to be aware of KV cache concatenation and selectively recomputing only part of the KV caches. Experiments on the RULER benchmark show that for a 124k‑token input, this combined method boosts the RULER score by 9.7 points over a baseline that recomputes caches only, while cutting TTFT by 80% compared with full attention.
By Fumihiko Tachibana, Daisuke Miyashita, Jun Deguchi
arXiv:2609.36722v1 Announce Type: new
Abstract: Large language model (LLM) agents repeatedly load reusable content, such as skills, documents, and memory entries, into the current context. Re-encodin...
By Xinghao Chen, Junnan Dong, Cai Ke, Chak Tou Leong, Haocheng Sun, Keyu Chen, Siyu An, Ruizhi Qiao, Xing Sun, Wenjie Li, Xiaoyu Shen