KV-Repurpose is an open‑source, vLLM‑based system that enables segment‑wise KV‑cache reuse beyond exact prompt prefixes in large language model serving. It integrates cache‑aware segment scheduling, prompt construction, capacity management, workload generation, and runtime monitoring to evaluate non‑prefix KV‑cache reuse. Experiments show that naive segment reuse maintains output quality for single‑document QA and summarization, while significantly improving cache hit rates and reducing time‑to‑first‑token compared to prefix caching, especially under Zipfian RAG‑style workloads.
By Taiga Ikeda, Daisuke Miyashita, Jun Deguchi
The paper investigates cache replacement strategies for large language model (LLM) prefix reuse, analyzing production traces from two companies and testing 14 eviction algorithms in both high-bandwidth memory (HBM) and large memory-pool environments. It finds that sophisticated policies designed for traditional caches offer little advantage over simple LRU, because prefix reuse is largely driven by the regular pacing of active sessions, making recency a strong predictor. The study also highlights new challenges such as heavy-tailed session footprints and variable miss costs, and proposes a compute-savings ratio along with two offline oracles to better quantify these effects, suggesting that effective prefix-cache management should combine recency with selective quick demotion, compute-aware partial eviction, and capacity-dependent granularity.
By Yiyu Liu, Minlan Yu, Juncheng Yang
arXiv:2608. 15127v1 Announce Type: cross Abstract: Agentic applications are shifting AI serving from isolated model inference to long-running workloads in which LLMs coordinate tools, environments, and persistent state.
By Chaokun Chang, Yukun Zhou, Kaihua Fu, Dakai An, Tianyu Feng, Hanfeng Lu, Sheng Yao, Pu Guo, Yinghao Yu, Yizhou Shan, Bo Li, Binhang Yuan, Wei Wang
The paper investigates how prefix caching, a default optimization in open‑source LLM serving stacks, affects reproducibility when combined with weight quantization. Experiments on an eighty‑episode multi‑turn agentic tool‑use workload show that enabling the cache causes the agent’s trajectory to change in 36.2 % of episodes at 16‑bit precision and 75.0 % at 4‑bit precision, while disabling the cache yields perfectly reproducible runs. The study identifies specific cache‑related settings that drive run‑to‑run divergence and demonstrates that cached serving is deterministic only when the cache state is preserved, which is not the case in typical deployments.
By Aditi Patodiya
CacheReforge is a method for recovering stale key‑value (KV) caches in large language models when lightweight adapters evolve. It represents stale caches as layer‑wise mixed‑version objects and uses adapter anchors, sensitivity calibration, drift accumulation, and restart boundaries to decide between direct reuse, bounded recomputation, or full suffix recovery. Experiments on Qwen2.5 models with continual LoRA updates show a 92.4% reduction in mean KL divergence while only recomputing 5.44% of layers and cutting cache‑maintenance time by 93.2% compared to full prefill.
By Yuhang Cao, Yanzhou Mu, Chunrong Fang, Zhenyu Chen
KV-Repurpose is an open‑source, vLLM‑based system that enables segment‑wise KV‑cache reuse beyond exact prompt prefixes in large language model serving. It integrates cache‑aware segment scheduling, prompt construction, capacity management, workload generation, and runtime monitoring. Experiments show that while naive segment reuse maintains output quality for single‑document tasks, it reveals limitations in cross‑document reasoning, and under a Zipfian RAG‑style workload it achieves a 63.9% token‑level cache hit rate versus 3.3% for prefix caching at 30 GB, dramatically lowering time‑to‑first‑token.