arXiv:2606. 09916v1 Announce Type: cross Abstract: Multi-turn LLM agents fan short queries into long trajectories of tool calls, search results, and intermediate reasoning.
arXiv:2609.14872v1 Announce Type: new Abstract: Agentic serving can consume orders of magnitude more tokens than chatbot workloads, stressing both KV-cache capacity and decode-time bandwidth. Most KV...
arXiv:2608. 07855v1 Announce Type: new Abstract: Multi-turn Reasoning-and-Acting (ReAct) agents accumulate growing trajectories of reasoning, tool calls, and observations.
arXiv:2609.03949v2 Announce Type: replace-cross Abstract: A long-lived KV cache must be compressed before the queries that will read it exist. Selection by observed attention collapses there: on a No...
arXiv:2607. 10582v1 Announce Type: cross Abstract: Large language model (LLM) agents accumulate heterogeneous context, including system instructions, plans, user turns, retrieved documents, tool outputs, and intermediate reasoning, whose key-value (KV) cache can become a major memory bottleneck.
VestigeKV is a new KV‑cache technique that uses a 64‑dimensional vestigial branch—originally a RoPE component repurposed during NoPE training—as a query‑independent eviction signal. By reading only 11 % of each cache row, the method partitions the cache into an attended tier (top‑m rows) and an archive tier (all other rows), which is GPU‑resident and never deleted. The approach achieves near‑perfect retrieval (1.00 at 8×, 0.92 at 32×) without any training, quantization, or changes to weights or kernels.
The paper investigates how prefix caching, a default optimization in open‑source LLM serving stacks, affects reproducibility when combined with weight quantization. Experiments on an eighty‑episode multi‑turn agentic tool‑use workload show that enabling the cache causes the agent’s trajectory to change in 36.2 % of episodes at 16‑bit precision and 75.0 % at 4‑bit precision, while disabling the cache yields perfectly reproducible runs. The study identifies specific cache‑related settings that drive run‑to‑run divergence and demonstrates that cached serving is deterministic only when the cache state is preserved, which is not the case in typical deployments.
arXiv:2606. 13126v1 Announce Type: cross Abstract: Retrieval-augmented and agentic workloads repeatedly prefill recurring predictable structured inputs (which we call "spans") such as documents and code files.
The paper introduces EpiKV, an epiphany‑aware key–value cache eviction strategy that avoids using the attention matrix. It leverages hidden‑state shifts and recent query–key relevance to rank cached tokens, matching or surpassing the performance of existing attention‑based eviction methods while remaining compatible with fast inference kernels. Experiments on multiple benchmarks show that EpiKV improves inference throughput without sacrificing accuracy.
arXiv:2608.21362v1 Announce Type: new Abstract: Transformer-based large language models (LLMs) incur high prefill latency because key-value (KV) tensors must be recomputed for each request. Existing...
Long-context large language model (LLM) inference is increasingly constrained by the memory footprint and decoding cost of key-value (KV) caches, limiting sustainable deployment on resource-constrained hardware. Existing KV cache eviction methods typically apply heuristic token scoring over all heads in GQA-based LLMs.
arXiv:2606. 24467v1 Announce Type: new Abstract: Long-context large language model (LLM) inference is increasingly constrained by the memory footprint and decoding cost of key-value (KV) caches, limiting sustainable deployment on resource-constrained hardware.
StepKV introduces a step-aware approach to compressing the key-value cache used during large language model inference, treating reasoning steps as primary units of retention rather than individual tokens. By linking cache entries to the steps that generated them and estimating each step’s utility from trajectory signals, StepKV assigns a combined token‑ and step‑level score to guide pruning. Experiments on multi‑hop question answering and long‑horizon web reasoning show that StepKV maintains accuracy even under tight cache budgets, outperforming token‑level baselines that suffer sharp performance drops.