← Back to all news
arXiv AI By Xingyu Qu, Siyuan Lu, Zhiyu Chen, Sheng Wang, Tao Lin

CacheBridge: Efficient Cross-Model KV Cache Transfer

Read the original on arXiv AI →

CacheBridge is a method for efficiently transferring key‑value (KV) caches between large language models (LLMs) in a multi‑model system. It replaces the full‑head mapping approach by matching each target KV head to a single source head, weighting reconstruction errors by causal attention sensitivity, and building weighted sufficient statistics with a fused GPU kernel. The technique achieves comparable or better accuracy to full‑head mapping while reducing mapper storage by up to eight‑fold, accelerating application by up to three‑times, and cutting construction time dramatically.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Oct 9

RaReCache: Bridging the Gap in Cross-Model KV Cache Reuse via Rank disagreement-based Selective Recomputation

RaReCache is a framework that allows a large target language model to decode accurately from a KV cache prefilled by a much smaller source model by selectively recomputing only critical token positions. It identifies these positions using a rank disagreement metric that scores tokens based on the energy of their mapped KV in output directions weakly supported by calibration data. Across two model families and five benchmarks, RaReCache achieves up to a 3.04× prefill speedup and significantly reduces prefill latency while maintaining 95–99% of target accuracy with a 30–40% recompute budget.

By Sreetama Sarkar, Saptarshi Mitra, Sitao Huang, Souvik Kundu, Peter A. Beerel
llmsagentsefficiencybenchmarks
More like this →
arXiv Machine Learning
Aug 5

Cross-Model KV Cache Transfer in LLM Families: A Closed-Form Linear Mapping for Prefill Reuse

arXiv:2608. 03893v1 Announce Type: new Abstract: Production deployments often swap between different-sized models in a family for cost-quality cascading, mid-conversation switching, and routing, and each swap forces the receiver to repay the prefill from scratch.

By Taekyung Heo, Rasoul Shafipour, Ritchie Zhao, Maximilian Golub, Mohammad Mahdi Kamani, Ritika Borkar, Makesh Tarun Chandran, Pantea Zardoshti, Bita Darvish Rouhani
llmsefficiency
More like this →