arXiv Machine Learning
Jun 19
Closing the Calibration Gap in Semantic Caching
arXiv:2606. 19719v1 Announce Type: cross Abstract: Semantic caching cuts LLM inference costs by serving a cached response to semantically similar queries.
Closing the Operational Gap in Semantic Caching
Semantic caching reduces LLM inference costs by returning cached responses for semantically similar queries, but current evaluation using PR‑AUC only ranks scores and ignores usability at a fixed threshold, leading to poor deployment choices. The authors propose a cache‑aware metric, Precision–Cache Hit Ratio (P‑CHR) AUC, and an Operational Retention Rate (ORR) to measure how offline ranking quality translates to deployment. They decompose the operational gap into a recoverable threshold‑utility component and an irreducible structural component, showing that the gap is driven by the training objective rather than data scale and can be mitigated by score re‑normalization or objective changes, framing model selection as a threshold‑utility problem.
Related stories
arXiv AI
Jul 7
Risk-Constrained Freshness-Aware Semantic Caching for Open-Web Retrieval-Augmented LLMs
arXiv:2607. 04281v1 Announce Type: cross Abstract: Semantic caching reduces the latency and cost of retrieval-augmented generation (RAG) by serving cached answers to semantically similar queries, but most existing methods do not model the time-varying freshness of open-web evidence.
arXiv AI
Jul 24
MiniCache: Reusable Program Caching with Small Model Interfaces for Efficient LLM Inference
arXiv:2607. 20507v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used for program-aided reasoning, agentic decision making, and structured task execution, but these applications often incur high inference cost.
arXiv Machine Learning
Jun 30
Diagnosing and Mitigating Retrieval Bottlenecks in LLM-Based Cold-Start Recommendation
arXiv:2606. 29947v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used as rerankers in recommender systems, with the expectation that semantic understanding will help in cold-start and long-tail regimes.
arXiv AI
Jul 20
Cache-Aware Prompt Compression:A Two-Tier Cost Model for LLM API Caching
arXiv:2607. 15516v1 Announce Type: cross Abstract: Production LLM deployments combine two cost-reduction primitives: prompt caching (a discounted rate for re-used token prefixes) and prompt compression (fewer tokens sent).
arXiv Machine Learning
Jun 5
Query-efficient model evaluation using cached responses
arXiv:2605. 07096v2 Announce Type: replace Abstract: Evaluating a new model on an existing benchmark is often necessary to understand its behavior before deployment.
arXiv AI
Jun 9
Semantic Cache Distillation: Efficient State Transfer via Reuse and Selective Patching
arXiv:2606. 07684v1 Announce Type: cross Abstract: Disaggregated serving alleviates memory bottlenecks in Large Language Model (LLM) inference but creates a severe communication bottleneck: transmitting high-dimensional Key-Value (KV) caches often dominates time-to-first-token (TTFT).
arXiv Machine Learning
Jul 20
VarRate: Training-Free Variable-Rate KV Cache Compression for Long-Context LLMs
arXiv:2607. 15498v1 Announce Type: cross Abstract: The key-value (KV) cache is the main memory bottleneck in long-context large language model (LLM) inference.
arXiv Machine Learning
Aug 24
From a Static Multi-Level Small Semantic Codebook to a Dynamic Single-Level Large Semantic Codebook for Generative Recommendation
The paper proposes replacing a static multi‑level small semantic codebook with a dynamic single‑level large semantic codebook for generative recommendation. It introduces an exposure‑aware update mechanism and an offline evaluation framework, achieving significant improvements in recall, NDCG, decoding efficiency, and online consumption metrics on public datasets and production traffic.
arXiv Computation and Language
4d ago
ATTUNER: Recomputation-Free KV Cache Reuse via Query-Side Adaptation
arXiv:2609.36722v1 Announce Type: new Abstract: Large language model (LLM) agents repeatedly load reusable content, such as skills, documents, and memory entries, into the current context. Re-encodin...
arXiv AI
Aug 14
LipCache: A Local Inference Proxy with Certified Caching for Edge Image Classification Service
arXiv:2608. 13144v1 Announce Type: cross Abstract: As edge-side vision services continue to expand toward low-latency, high-throughput scenarios, reducing the inference cost of vision models without sacrificing reliability has become a central concern.
arXiv AI
Aug 25
CacheSpec: Finding the Sweet Spot for Small Models in Large Language Models
CacheSpec is an inference optimization framework that transforms Program-of-Thoughts (PoT) style programs into reusable cache objects for large language models. By employing a small model for semantic variable extraction on cache hits and speculative drafting during target-LLM generation, CacheSpec reduces inference latency and improves cache reuse. Experiments on shopping, web, formula, and code QA datasets demonstrate up to 3.1× speedup in latency and 2.8× throughput gains over traditional PoT methods, while maintaining or improving task quality.