arXiv AI
A Universal Context-Reuse Layer for Cross-Model KV Sharing
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Oct 9
RaReCache: Bridging the Gap in Cross-Model KV Cache Reuse via Rank disagreement-based Selective Recomputation
RaReCache is a framework that allows a large target language model to decode accurately from a KV cache prefilled by a much smaller source model by selectively recomputing only critical token positions. It identifies these positions using a rank disagreement metric that scores tokens based on the energy of their mapped KV in output directions weakly supported by calibration data. Across two model families and five benchmarks, RaReCache achieves up to a 3.04× prefill speedup and significantly reduces prefill latency while maintaining 95–99% of target accuracy with a 30–40% recompute budget.
arXiv AI
6d ago
CacheLink: Functional KV Transport for Zero-Prefill Cross-Model Context Handoff
arXiv:2610.15509v1 Announce Type: cross Abstract: KV caches make long-context serving efficient by reusing past computation, but model switching breaks this reuse and forces the receiver to prefill t...
arXiv AI
Oct 1
Prefill-Free Cross-Family KV Cache Transfer for Heterogeneous Multi-Agent LLMs
arXiv:2609.32259v2 Announce Type: replace Abstract: Recent multi-agent LLM systems increasingly combine heterogeneous models for specialized agent roles. However, text-based communication requires ea...
arXiv AI
Sep 16
Shared-Prefix KV Reuse Across Standard LoRA Adapters: Quality and Serving Tradeoffs
arXiv:2609.17109v1 Announce Type: new Abstract: A common small-model deployment runs one shared backbone with several LoRA specialists that answer over the same context. Serving them naively re-prefi...
arXiv Machine Learning
Sep 15
Physically Partitioned KVCache Format for CPU--GPU Load Balancing in MoE Inference
arXiv:2609.14507v1 Announce Type: cross Abstract: Single-GPU long-context inference with Mixture-of-Experts (MoE) models requires spilling the key-value cache (KVCache) to CPU memory. The spilled KV...
arXiv AI
Sep 3
HeadWiseKV: Budgeted Per-Head Cache Residency for Hybrid Long-Context Language Models
HeadWiseKV is a training‑free framework that compresses the residual global key–value caches of hybrid long‑context language models by assigning each physical KV head a static, multilevel history window. It formulates cache allocation as a restricted operational rate–distortion problem and uses the SeqCalib algorithm to generate per‑head residency policies that account for interactions across layers. In evaluations on four hybrid models, HeadWiseKV preserves near‑full‑KV quality while reducing peak device memory usage by 8.59% at a 112K context length and extending the largest verified context from 114K to 161K.