arXiv Machine Learning
Continuous Semantic Caching for Low-Cost LLM Serving
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Jun 30
Learning to Route and Schedule LLMs from User Retrials via Contextual Queueing Bandits
arXiv:2602. 02061v2 Announce Type: replace Abstract: Explosive demands for LLMs often cause user queries to accumulate in server queues, requiring efficient routing (query-LLM matching) and scheduling (query prioritization) mechanisms.
arXiv Machine Learning
Jun 5
Query-efficient model evaluation using cached responses
arXiv:2605. 07096v2 Announce Type: replace Abstract: Evaluating a new model on an existing benchmark is often necessary to understand its behavior before deployment.
arXiv AI
Jul 7
Risk-Constrained Freshness-Aware Semantic Caching for Open-Web Retrieval-Augmented LLMs
arXiv:2607. 04281v1 Announce Type: cross Abstract: Semantic caching reduces the latency and cost of retrieval-augmented generation (RAG) by serving cached answers to semantically similar queries, but most existing methods do not model the time-varying freshness of open-web evidence.
arXiv Machine Learning
Aug 31
Closing the Operational Gap in Semantic Caching
Semantic caching reduces LLM inference costs by returning cached responses for semantically similar queries, but current evaluation using PR‑AUC only ranks scores and ignores usability at a fixed threshold, leading to poor deployment choices. The authors propose a cache‑aware metric, Precision–Cache Hit Ratio (P‑CHR) AUC, and an Operational Retention Rate (ORR) to measure how offline ranking quality translates to deployment. They decompose the operational gap into a recoverable threshold‑utility component and an irreducible structural component, showing that the gap is driven by the training objective rather than data scale and can be mitigated by score re‑normalization or objective changes, framing model selection as a threshold‑utility problem.
arXiv AI
Oct 5
JOVE: Joint Execution and Verification for Resource-Aware LLM Task Graphs
JOVE is an online framework that jointly assigns large language model (LLM) executors and selects intermediate outputs for paid verification in directed acyclic task graphs. It balances immediate execution costs with future learning by solving per-query mixed‑integer linear programs and updating task‑dependent LLM quality estimates through verification feedback. Experiments on four reasoning benchmarks show that JOVE matches standard inference accuracy while cutting average cost and latency by at least 3.17×.
arXiv AI
Aug 11
TreeHop: Efficient Embedding-Level Query Rewriter
arXiv:2504. 20114v3 Announce Type: replace-cross Abstract: Retrieval-augmented generation (RAG) systems face significant challenges in multi-hop question answering (MHQA), where complex queries require synthesizing information across multiple document chunks.