arXiv AI
CacheRouter: A Dual-Path Tool Routing Architecture with Cache-Preserving Main-Model Isolation for Long-Tail Tool Discovery
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Machine Learning
Sep 7
Same Request, Different Answer: Quantization Amplifies Cache-Induced Divergence in LLM Serving
The paper investigates how prefix caching, a default optimization in open‑source LLM serving stacks, affects reproducibility when combined with weight quantization. Experiments on an eighty‑episode multi‑turn agentic tool‑use workload show that enabling the cache causes the agent’s trajectory to change in 36.2 % of episodes at 16‑bit precision and 75.0 % at 4‑bit precision, while disabling the cache yields perfectly reproducible runs. The study identifies specific cache‑related settings that drive run‑to‑run divergence and demonstrates that cached serving is deterministic only when the cache state is preserved, which is not the case in typical deployments.
arXiv Computation and Language
5d ago
Cache Engineering for Agents
The paper introduces cache engineering for large language model agents, enabling direct control over the active key-value cache without altering the trajectory or triggering costly re-prefilling. It defines cache operations such as drop and repos, formalizes their effects, and addresses the challenge of history-dependent cache states. The authors co-design a stateless AKV serving engine that manages stateful cache reuse, eviction, and restoration, achieving significant performance and cost improvements over traditional summarization methods.
arXiv AI
Jul 20
Cache-Aware Prompt Compression:A Two-Tier Cost Model for LLM API Caching
arXiv:2607. 15516v1 Announce Type: cross Abstract: Production LLM deployments combine two cost-reduction primitives: prompt caching (a discounted rate for re-used token prefixes) and prompt compression (fewer tokens sent).
arXiv AI
Aug 18
From LLM Inference to Agentic Workloads: Characterization and Implications for Serving Systems
arXiv:2608. 15127v1 Announce Type: cross Abstract: Agentic applications are shifting AI serving from isolated model inference to long-running workloads in which LLMs coordinate tools, environments, and persistent state.
arXiv AI
Aug 25
AgentWeave: Routing Before Reasoning for Efficient Function Calling in Tool-Rich Language Models
arXiv:2608.23078v1 Announce Type: new Abstract: Large language models increasingly operate over large collections of tools, functions, APIs, and specialized agents. As the candidate action space grow...
arXiv AI
Aug 26
Hybrid Semantic Tool Discovery for Enterprise MCP Gateway: Architecture and Implementation
Hybrid Semantic Tool Discovery for Enterprise MCP Gateway presents SCOUT, a system that addresses two major challenges in large language model (LLM) agent tool usage: a context‑engineering bottleneck and a tool discoverability barrier. SCOUT reframes tool exposure as a context‑selection problem, injecting only relevant tools into the model’s context window and providing two MCP meta‑tools—tool_search and execute_tool—to perform hybrid retrieval via BM25 and dense vector search. In production at PayPal, SCOUT cuts MCP tool‑token consumption by 99%, dramatically reducing per‑query inference cost while remaining model‑agnostic and requiring no client‑side changes.