← Back to all news
arXiv AI By Sreetama Sarkar, Saptarshi Mitra, Sitao Huang, Souvik Kundu, Peter A. Beerel

RaReCache: Bridging the Gap in Cross-Model KV Cache Reuse via Rank disagreement-based Selective Recomputation

Read the original on arXiv AI →

RaReCache is a framework that allows a large target language model to decode accurately from a KV cache prefilled by a much smaller source model by selectively recomputing only critical token positions. It identifies these positions using a rank disagreement metric that scores tokens based on the energy of their mapped KV in output directions weakly supported by calibration data. Across two model families and five benchmarks, RaReCache achieves up to a 3.04× prefill speedup and significantly reduces prefill latency while maintaining 95–99% of target accuracy with a 30–40% recompute budget.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Sep 2

CacheBridge: Efficient Cross-Model KV Cache Transfer

CacheBridge is a method for efficiently transferring key‑value (KV) caches between large language models (LLMs) in a multi‑model system. It replaces the full‑head mapping approach by matching each target KV head to a single source head, weighting reconstruction errors by causal attention sensitivity, and building weighted sufficient statistics with a fused GPU kernel. The technique achieves comparable or better accuracy to full‑head mapping while reducing mapper storage by up to eight‑fold, accelerating application by up to three‑times, and cutting construction time dramatically.

By Xingyu Qu, Siyuan Lu, Zhiyu Chen, Sheng Wang, Tao Lin
llmsefficiency
More like this →
arXiv Machine Learning
Aug 5

Cross-Model KV Cache Transfer in LLM Families: A Closed-Form Linear Mapping for Prefill Reuse

arXiv:2608. 03893v1 Announce Type: new Abstract: Production deployments often swap between different-sized models in a family for cost-quality cascading, mid-conversation switching, and routing, and each swap forces the receiver to repay the prefill from scratch.

By Taekyung Heo, Rasoul Shafipour, Ritchie Zhao, Maximilian Golub, Mohammad Mahdi Kamani, Ritika Borkar, Makesh Tarun Chandran, Pantea Zardoshti, Bita Darvish Rouhani
llmsefficiency
More like this →
arXiv Machine Learning
Sep 29

CacheReforge: Bounded Recovery for Stale KV Caches under Evolving Adapters

CacheReforge is a method for recovering stale key‑value (KV) caches in large language models when lightweight adapters evolve. It represents stale caches as layer‑wise mixed‑version objects and uses adapter anchors, sensitivity calibration, drift accumulation, and restart boundaries to decide between direct reuse, bounded recomputation, or full suffix recovery. Experiments on Qwen2.5 models with continual LoRA updates show a 92.4% reduction in mean KL divergence while only recomputing 5.44% of layers and cutting cache‑maintenance time by 93.2% compared to full prefill.

By Yuhang Cao, Yanzhou Mu, Chunrong Fang, Zhenyu Chen
llmsfine-tuning
More like this →