WAKMan is a workload‑aware KV cache manager designed for Large Language Model servers that selects an eviction policy based on live traffic patterns without needing to know request types or read prompts. By adapting to the mix of single‑turn, multi‑turn, and agentic requests, it improves cache hit rates over the default LRU strategy by an average of 5 percentage points and achieves the lowest mean and worst‑case optimality gap among evaluated policies. The paper demonstrates that no single policy consistently outperforms others across diverse traffic mixes, highlighting the need for dynamic, workload‑aware cache management.
By Suhas Kumar Morisetty, Vaibhavi Lokegaonkar, Sachin Mathew Varghese, Wesley Harris
arXiv:2608. 14624v1 Announce Type: new Abstract: Multi-agent LLM systems have emerged as an important deployment paradigm for AI services, where each user request is decomposed into a sequence of specialized agents.
By Rui Zhang, Chaeeun Kim, Shaoting Feng, Kuntai Du, Yuhan Liu, Yi Zhong, Cheng-Wei Ching, Junchen Jiang, Liting Hu
The paper investigates cache replacement strategies for large language model (LLM) prefix reuse, analyzing production traces from two companies and testing 14 eviction algorithms in both high-bandwidth memory (HBM) and large memory-pool environments. It finds that sophisticated policies designed for traditional caches offer little advantage over simple LRU, because prefix reuse is largely driven by the regular pacing of active sessions, making recency a strong predictor. The study also highlights new challenges such as heavy-tailed session footprints and variable miss costs, and proposes a compute-savings ratio along with two offline oracles to better quantify these effects, suggesting that effective prefix-cache management should combine recency with selective quick demotion, compute-aware partial eviction, and capacity-dependent granularity.
By Yiyu Liu, Minlan Yu, Juncheng Yang
arXiv:2609.14872v1 Announce Type: new
Abstract: Agentic serving can consume orders of magnitude more tokens than chatbot workloads, stressing both KV-cache capacity and decode-time bandwidth. Most KV...
By Taowen Tony Liu, Jeffrey T. H. Wong, Can Xiao, Bowen Yang, Hao Mark Chen, Yiren Zhao
The paper introduces cache engineering for large language model agents, enabling direct control over the active key-value cache without altering the trajectory or triggering costly re-prefilling. It defines cache operations such as drop and repos, formalizes their effects, and addresses the challenge of history-dependent cache states. The authors co-design a stateless AKV serving engine that manages stateful cache reuse, eviction, and restoration, achieving significant performance and cost improvements over traditional summarization methods.
By Tianyu Fu, Ruoxi Wang, Zihan Min, Zhongqi Ren, Zhihao Li, Tengxuan Liu, Shengxu Mu, Hanling Zhang, Yu Wang
arXiv:2608. 06557v1 Announce Type: cross Abstract: The reasoning and agentic capabilities of large language models have expanded the range of applications they support, from short interactive exchanges to long, compute-heavy requests.
By Muhammad Adnan, Rohan Mahapatra, Prashant J. Nair, Daniel Berger, Pantea Zardoshti, Rodrigo Fonseca, Esha Choukse
arXiv:2607. 02525v1 Announce Type: cross Abstract: We present PEEK, a lightweight scheduling and eviction framework for both online (streaming) and offline (batch) LLM serving; this paper focuses on the online regime.
By Bing Xie, Zhipeng Wang, Masahiro Tanaka, Zheng Zhen
TOPAS is a Task‑Oriented Prefix‑Aware Scheduler designed for multi‑agent large language model serving. It jointly decides which agent prefixes to retain in a shared key‑value cache and which requests to schedule, balancing the reduction of each task’s longest remaining service path against the benefit of downstream prefix reuse while accounting for movement and preemption costs. Experiments on synthetic DAGs and MetaGPT software‑development workflows show that TOPAS can reduce mean and p99 job completion times by up to 39.8%/49.4% and 22.0%/26.6% respectively compared to the best baselines.
By Hongqiu Ni, Han Tian, Chi Zhang, Guopeng Li, Haisheng Tan
arXiv:2609.34060v2 Announce Type: replace
Abstract: Prompt-specialized multi-agent systems enable multiple agents to share a model while performing complementary roles to solve complex tasks. However...
By Hyesung Jeon, Hyeongju Ha, Seoyoung Lee, Beomseok Kang, Jae-Joon Kim
KV-Repurpose is an open‑source, vLLM‑based system that enables segment‑wise KV‑cache reuse beyond exact prompt prefixes in large language model serving. It integrates cache‑aware segment scheduling, prompt construction, capacity management, workload generation, and runtime monitoring to evaluate non‑prefix KV‑cache reuse. Experiments show that naive segment reuse maintains output quality for single‑document QA and summarization, while significantly improving cache hit rates and reducing time‑to‑first‑token compared to prefix caching, especially under Zipfian RAG‑style workloads.
By Taiga Ikeda, Daisuke Miyashita, Jun Deguchi
arXiv:2606. 09916v1 Announce Type: cross Abstract: Multi-turn LLM agents fan short queries into long trajectories of tool calls, search results, and intermediate reasoning.
By Junjie Li, Jiong Lou, Jie Li
arXiv:2604.05012v2 Announce Type: replace-cross
Abstract: Efficient inference with Large Language Models (LLMs) increasingly relies on Key-Value (KV) caches to store previously computed key and value...
By Oteo Mamo, Olga Kogiou, Hyunjin Yi, Weikuan Yu