arXiv:2607. 04281v1 Announce Type: cross Abstract: Semantic caching reduces the latency and cost of retrieval-augmented generation (RAG) by serving cached answers to semantically similar queries, but most existing methods do not model the time-varying freshness of open-web evidence.
By Muhammad Mansoor, Tahir Ahmad, Yeo-Chan Yoon
arXiv:2608.30647v1 Announce Type: cross
Abstract: Language models can answer from precomputed memory, a model's saved reading of a body of material, reused across requests instead of read again at ea...
By Asa Shepard
EntropyCache is a training‑free KV caching technique for diffusion‑based large language models that uses the maximum entropy of newly decoded token distributions to decide when to recompute cached states. By leveraging two empirical observations—entropy correlating with cache drift and feature volatility persisting across multiple steps—EntropyCache makes a skip‑or‑recompute decision with only O(V) computation per step, independent of context length or model size. Experiments on LLaDA‑8B‑Instruct and Dream‑7B‑Instruct demonstrate speedups of 15.2×–26.4× on standard benchmarks and 22.4×–24.1× on chain‑of‑thought benchmarks while maintaining competitive accuracy and keeping decision overhead below 0.5% of inference time.
By Minsoo Cheong, Donghyun Son, Woosang Lim, Sungjoo Yoo
Semantic caching reduces LLM inference costs by returning cached responses for semantically similar queries, but current evaluation using PR‑AUC only ranks scores and ignores usability at a fixed threshold, leading to poor deployment choices. The authors propose a cache‑aware metric, Precision–Cache Hit Ratio (P‑CHR) AUC, and an Operational Retention Rate (ORR) to measure how offline ranking quality translates to deployment. They decompose the operational gap into a recoverable threshold‑utility component and an irreducible structural component, showing that the gap is driven by the training objective rather than data scale and can be mitigated by score re‑normalization or objective changes, framing model selection as a threshold‑utility problem.
By Aditeya Baral, Radoslav Ralev, Iliya Sotirov Zhechev, Srijith Rajamohan, Jen Agarwal
arXiv:2607. 15516v1 Announce Type: cross Abstract: Production LLM deployments combine two cost-reduction primitives: prompt caching (a discounted rate for re-used token prefixes) and prompt compression (fewer tokens sent).
By Yan Song
arXiv:2606. 19719v1 Announce Type: cross Abstract: Semantic caching cuts LLM inference costs by serving a cached response to semantically similar queries.
By Aditeya Baral, Radoslav Ralev, Iliya Sotirov Zhechev, Srijith Rajamohan, Jen Agarwal