arXiv AI August 20, 2026 By Sukanta Ganguly

Fractional Decay KV-Cache: Ownership-Aware Memory Management for Improved Inference Relevancy in Dialog Systems

Fractional Decay KV-Cache (FD‑KVC) is a new memory‑management algorithm for transformer‑based dialog systems that uses a dual‑channel scoring mechanism—cumulative attention and recency‑weighted relevance—to keep key‑value pairs relevant as conversation topics shift. The method incorporates an ownership‑loss‑driven adaptive learning rate, runs entirely on CPU with minimal overhead, and outperforms the heavy‑hitter baseline H2O by significant margins across multiple dialog scenarios, including faster topic adaptation and higher topic diversity. Ablation studies confirm that each component of FD‑KVC contributes to its performance gains.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI

Jul 14

MemDecay: Region-Aware KV Cache Eviction for Efficient LLM Agent Inference

arXiv:2607. 10582v1 Announce Type: cross Abstract: Large language model (LLM) agents accumulate heterogeneous context, including system instructions, plans, user turns, retrieved documents, tool outputs, and intermediate reasoning, whose key-value (KV) cache can become a major memory bottleneck.

By Venkatesha Matam, Keon Kim

llms agents efficiency

arXiv Machine Learning

Jun 29

Learning to Evict from Key-Value Cache

arXiv:2602. 10238v2 Announce Type: replace-cross Abstract: The growing size of Large Language Models (LLMs) makes efficient inference challenging, primarily due to the memory demands of the autoregressive Key-Value (KV) cache.

By Luca Moschella, Laura Manduchi, Ozan Sener

llms agents reinforcement-learning efficiency benchmarks

arXiv Machine Learning

Aug 7

QEvict: Recoverable Quantized KV Eviction for Attention-Drift-Robust Long-Context Decoding

arXiv:2608. 05326v1 Announce Type: new Abstract: Autoregressive large language model inference is increasingly constrained by the memory footprint of the Key-Value (KV) cache.

By Ayushman Garg, Akshita Gupta, Shaswata Bhattacharya, Abhishek Gupta, Sandeep Kumar, Manoj Kumar

llms efficiency benchmarks

arXiv AI

Oct 7

AttSVD:Prompt-Adaptive Low-Rank KV Cache Compression via Attention-Guided SVD

arXiv:2610.06927v1 Announce Type: cross Abstract: The key-value (KV) cache of autoregressive transformers grows linearly with context length and dominates memory at long context. Most training-free r...

By Sara Abdali, Jongwoo Ko, Pashmina Cameron

llms agents efficiency benchmarks safety

arXiv Computation and Language

2d ago

Reformulating KV Cache Eviction Problem for Long-Context LLM Inference

The paper proposes LaProx, a new KV cache eviction strategy for long‑context LLM inference that redefines the problem as an output‑aware, layer‑wise matrix multiplication approximation. Unlike prior methods that rely only on local attention weights, LaProx explicitly models the interaction between attention maps and projected value states, accounting for inter‑head dependencies to assign globally comparable token importance scores. Experiments on 19 datasets, including LongBench and Needle‑In‑A‑Haystack, show that LaProx retains model performance while using only 5% of the KV cache and reduces accuracy loss by up to 2× under extreme compression compared to state‑of‑the‑art baselines.

By Tho Mai, Joo-Young Kim

llms efficiency benchmarks

Hugging Face Trending Papers

Oct 5

DeferKV: Rethinking Eviction Timing for One-Shot KV Cache Compression

DeferKV rethinks when to evict key‑value (KV) cache entries in long‑context large language models. By delaying eviction until the first decoding step and combining prompt‑side and decode‑side attention signals, it aligns KV importance estimation with actual generation needs. The method requires no extra training or modules and consistently improves performance on benchmarks while keeping latency low.

llms efficiency