StepKV introduces a step-aware approach to compressing the key-value cache used during large language model inference, treating reasoning steps as primary units of retention rather than individual tokens. By linking cache entries to the steps that generated them and estimating each step’s utility from trajectory signals, StepKV assigns a combined token‑ and step‑level score to guide pruning. Experiments on multi‑hop question answering and long‑horizon web reasoning show that StepKV maintains accuracy even under tight cache budgets, outperforming token‑level baselines that suffer sharp performance drops.
arXiv:2602. 10238v2 Announce Type: replace-cross Abstract: The growing size of Large Language Models (LLMs) makes efficient inference challenging, primarily due to the memory demands of the autoregressive Key-Value (KV) cache.
arXiv:2602. 03203v2 Announce Type: replace-cross Abstract: Recently, large language models (LLMs) have shown remarkable reasoning abilities by producing long reasoning traces.
arXiv:2608. 04771v1 Announce Type: new Abstract: Large Reasoning Models (LRMs) excel on complex tasks through long chain-of-thought (CoT) reasoning, but their lengthy intermediate steps cause severe overthinking that inflates inference cost.
arXiv:2608. 07855v1 Announce Type: new Abstract: Multi-turn Reasoning-and-Acting (ReAct) agents accumulate growing trajectories of reasoning, tool calls, and observations.
arXiv:2607. 10582v1 Announce Type: cross Abstract: Large language model (LLM) agents accumulate heterogeneous context, including system instructions, plans, user turns, retrieved documents, tool outputs, and intermediate reasoning, whose key-value (KV) cache can become a major memory bottleneck.
BeaconKV is a training‑free key‑value cache compression technique for Large Reasoning Models that uses beacon queries—compact representatives of query clusters—to predict which KV pairs will be revisited during long‑horizon reasoning. By focusing on Thought Revisiting Tokens that re‑attend distant context, BeaconKV reduces memory usage up to 5.8× and improves throughput by over 4.3× while largely preserving cache accuracy across multiple open‑source LRMs and reasoning benchmarks.
Large Reasoning Models (LRMs) excel on complex tasks through long chain-of-thought (CoT) reasoning, but their lengthy intermediate steps cause severe overthinking that inflates inference cost. KV-cache compression is a common solution, yet existing reasoning-oriented methods apply a uniform policy across the trajectory and judge compression only by what it removes from the cache.
arXiv:2508. 15806v2 Announce Type: replace-cross Abstract: The growing sequence length of large language models poses significant challenges for key-value (KV) caches.
arXiv:2606. 26875v1 Announce Type: cross Abstract: Reasoning capability has advanced rapidly in large language models (LLMs), leading to an increasing size of key-value (KV) cache in both prefilling and decoding stages.
arXiv:2606. 24467v1 Announce Type: new Abstract: Long-context large language model (LLM) inference is increasingly constrained by the memory footprint and decoding cost of key-value (KV) caches, limiting sustainable deployment on resource-constrained hardware.
Long-context large language model (LLM) inference is increasingly constrained by the memory footprint and decoding cost of key-value (KV) caches, limiting sustainable deployment on resource-constrained hardware. Existing KV cache eviction methods typically apply heuristic token scoring over all heads in GQA-based LLMs.
arXiv:2608. 15797v1 Announce Type: new Abstract: KV-cache eviction caps the memory cost of long reasoning traces but is inherently lossy because the model decodes from a partial view of its history.