Cache Engineering for Agents

The paper introduces cache engineering for large language model agents, enabling direct control over the active key-value cache without altering the trajectory or triggering costly re-prefilling. It defines cache operations such as drop and repos, formalizes their effects, and addresses the challenge of history-dependent cache states. The authors co-design a stateless AKV serving engine that manages stateful cache reuse, eviction, and restoration, achieving significant performance and cost improvements over traditional summarization methods.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

• llms
• agents
• nlp
• efficiency
• benchmarks