Hugging Face Trending Papers
WAKMan: Workload Aware KVCache Manager
Read the original on Hugging Face Trending Papers →The Flow has not summarised this story yet — read it at Hugging Face Trending Papers.
Related stories
arXiv AI
6d ago
WAKMan: Workload Aware KVCache Manager
WAKMan is a workload‑aware KV cache manager designed for Large Language Model servers that selects an eviction policy based on live traffic patterns without needing to know request types or read prompts. By adapting to the mix of single‑turn, multi‑turn, and agentic requests, it improves cache hit rates over the default LRU strategy by an average of 5 percentage points and achieves the lowest mean and worst‑case optimality gap among evaluated policies. The paper demonstrates that no single policy consistently outperforms others across diverse traffic mixes, highlighting the need for dynamic, workload‑aware cache management.
arXiv AI
Aug 18
Learning Agent Execution for KV-Cache Management in Agentic Serving
arXiv:2608. 14624v1 Announce Type: new Abstract: Multi-agent LLM systems have emerged as an important deployment paradigm for AI services, where each user request is decomposed into a sequence of specialized agents.
arXiv Machine Learning
Sep 25
When Fancy Eviction Fails: Rethinking Cache Replacement For LLM Prefix Reuse
The paper investigates cache replacement strategies for large language model (LLM) prefix reuse, analyzing production traces from two companies and testing 14 eviction algorithms in both high-bandwidth memory (HBM) and large memory-pool environments. It finds that sophisticated policies designed for traditional caches offer little advantage over simple LRU, because prefix reuse is largely driven by the regular pacing of active sessions, making recency a strong predictor. The study also highlights new challenges such as heavy-tailed session footprints and variable miss costs, and proposes a compute-savings ratio along with two offline oracles to better quantify these effects, suggesting that effective prefix-cache management should combine recency with selective quick demotion, compute-aware partial eviction, and capacity-dependent granularity.
arXiv Machine Learning
Sep 15
AgentKV: Phase-Aware KV Eviction for Agentic LLMs
arXiv:2609.14872v1 Announce Type: new Abstract: Agentic serving can consume orders of magnitude more tokens than chatbot workloads, stressing both KV-cache capacity and decode-time bandwidth. Most KV...
arXiv Computation and Language
Oct 21
Cache Engineering for Agents
The paper introduces cache engineering for large language model agents, enabling direct control over the active key-value cache without altering the trajectory or triggering costly re-prefilling. It defines cache operations such as drop and repos, formalizes their effects, and addresses the challenge of history-dependent cache states. The authors co-design a stateless AKV serving engine that manages stateful cache reuse, eviction, and restoration, achieving significant performance and cost improvements over traditional summarization methods.
arXiv Machine Learning
Aug 10
Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving
arXiv:2608. 06557v1 Announce Type: cross Abstract: The reasoning and agentic capabilities of large language models have expanded the range of applications they support, from short interactive exchanges to long, compute-heavy requests.