arXiv:2606. 19719v1 Announce Type: cross Abstract: Semantic caching cuts LLM inference costs by serving a cached response to semantically similar queries.
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.
Semantic caching reduces LLM inference costs by returning cached responses for semantically similar queries, but current evaluation using PR‑AUC only ranks scores and ignores usability at a fixed threshold, leading to poor deployment choices. The authors propose a cache‑aware metric, Precision–Cache Hit Ratio (P‑CHR) AUC, and an Operational Retention Rate (ORR) to measure how offline ranking quality translates to deployment. They decompose the operational gap into a recoverable threshold‑utility component and an irreducible structural component, showing that the gap is driven by the training objective rather than data scale and can be mitigated by score re‑normalization or objective changes, framing model selection as a threshold‑utility problem.
arXiv:2607. 04281v1 Announce Type: cross Abstract: Semantic caching reduces the latency and cost of retrieval-augmented generation (RAG) by serving cached answers to semantically similar queries, but most existing methods do not model the time-varying freshness of open-web evidence.
arXiv:2607. 20507v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used for program-aided reasoning, agentic decision making, and structured task execution, but these applications often incur high inference cost.
arXiv:2605. 07096v2 Announce Type: replace Abstract: Evaluating a new model on an existing benchmark is often necessary to understand its behavior before deployment.
arXiv:2607. 15516v1 Announce Type: cross Abstract: Production LLM deployments combine two cost-reduction primitives: prompt caching (a discounted rate for re-used token prefixes) and prompt compression (fewer tokens sent).
arXiv:2609.36722v1 Announce Type: new Abstract: Large language model (LLM) agents repeatedly load reusable content, such as skills, documents, and memory entries, into the current context. Re-encodin...