arXiv Machine Learning September 22, 2026 By Boyu Feng, Jiahong Liu, Yifan Li, Wenhao Yu, Zexuan Qiu, Yuliang Sun, Ming Shen, Xiang Li, Quanyu Dai, Irwin King

StepKV: Step-Aware KV Cache Compression for LLM Agents

Read the original on arXiv Machine Learning →

StepKV introduces a step-aware approach to compressing the key-value cache used during large language model inference, treating reasoning steps as primary units of retention rather than individual tokens. By linking cache entries to the steps that generated them and estimating each step’s utility from trajectory signals, StepKV assigns a combined token‑ and step‑level score to guide pruning. Experiments on multi‑hop question answering and long‑horizon web reasoning show that StepKV maintains accuracy even under tight cache budgets, outperforming token‑level baselines that suffer sharp performance drops.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Machine Learning

Jun 29

Learning to Evict from Key-Value Cache

arXiv:2602. 10238v2 Announce Type: replace-cross Abstract: The growing size of Large Language Models (LLMs) makes efficient inference challenging, primarily due to the memory demands of the autoregressive Key-Value (KV) cache.

By Luca Moschella, Laura Manduchi, Ozan Sener

llms agents reinforcement-learning efficiency benchmarks

arXiv Machine Learning

Jun 2

ForesightKV: Optimizing KV Cache Eviction for Reasoning Models by Learning Long-Term Contribution

arXiv:2602. 03203v2 Announce Type: replace-cross Abstract: Recently, large language models (LLMs) have shown remarkable reasoning abilities by producing long reasoning traces.

By Zican Dong, Peiyu Liu, Junyi Li, Zhipeng Chen, Han Peng, Shuo Wang, Wayne Xin Zhao

llms reinforcement-learning efficiency benchmarks

arXiv AI

Aug 6

Fewer Tokens, Smaller Cache: Reward-Coordinated Efficient Reasoning

arXiv:2608. 04771v1 Announce Type: new Abstract: Large Reasoning Models (LRMs) excel on complex tasks through long chain-of-thought (CoT) reasoning, but their lengthy intermediate steps cause severe overthinking that inflates inference cost.

By Qiyuan Zhu, Dezhi Li, Pengyu Cheng, Tianle Chen, Jiacheng Wang, Ruijie Shen, Hao Gu, Sida Lin, Zirui Liu, Jiacheng Liu, Sirui Han

benchmarks

arXiv Machine Learning

Aug 11

CommitKV: Lifecycle-Aware KV Cache Compression via Commit Transitions for Multi-Turn Agents

arXiv:2608. 07855v1 Announce Type: new Abstract: Multi-turn Reasoning-and-Acting (ReAct) agents accumulate growing trajectories of reasoning, tool calls, and observations.

By Weizhong Huang, Jinchao Zhang, Xiawu Zheng

agents efficiency benchmarks

arXiv AI

Jul 14

MemDecay: Region-Aware KV Cache Eviction for Efficient LLM Agent Inference

arXiv:2607. 10582v1 Announce Type: cross Abstract: Large language model (LLM) agents accumulate heterogeneous context, including system instructions, plans, user turns, retrieved documents, tool outputs, and intermediate reasoning, whose key-value (KV) cache can become a major memory bottleneck.

By Venkatesha Matam, Keon Kim

llms agents efficiency

arXiv Machine Learning

Sep 7

BeaconKV: Key-Value Cache Compression Guided by Beacon Queries for Efficient Large Reasoning Model Inference

BeaconKV is a training‑free key‑value cache compression technique for Large Reasoning Models that uses beacon queries—compact representatives of query clusters—to predict which KV pairs will be revisited during long‑horizon reasoning. By focusing on Thought Revisiting Tokens that re‑attend distant context, BeaconKV reduces memory usage up to 5.8× and improves throughput by over 4.3× while largely preserving cache accuracy across multiple open‑source LRMs and reasoning benchmarks.

By Janghyeon Kim, Minsoo Kim, Kyuhong Shim, Jungwook Choi

rag efficiency benchmarks

One email a morning, machine-written

Related stories

Learning to Evict from Key-Value Cache

ForesightKV: Optimizing KV Cache Eviction for Reasoning Models by Learning Long-Term Contribution

Fewer Tokens, Smaller Cache: Reward-Coordinated Efficient Reasoning

CommitKV: Lifecycle-Aware KV Cache Compression via Commit Transitions for Multi-Turn Agents

MemDecay: Region-Aware KV Cache Eviction for Efficient LLM Agent Inference

BeaconKV: Key-Value Cache Compression Guided by Beacon Queries for Efficient Large Reasoning Model Inference