Fractional Decay KV-Cache (FD‑KVC) is a new memory‑management algorithm for transformer‑based dialog systems that uses a dual‑channel scoring mechanism—cumulative attention and recency‑weighted relevance—to keep key‑value pairs relevant as conversation topics shift. The method incorporates an ownership‑loss‑driven adaptive learning rate, runs entirely on CPU with minimal overhead, and outperforms the heavy‑hitter baseline H2O by significant margins across multiple dialog scenarios, including faster topic adaptation and higher topic diversity. Ablation studies confirm that each component of FD‑KVC contributes to its performance gains.
arXiv:2607. 10582v1 Announce Type: cross Abstract: Large language model (LLM) agents accumulate heterogeneous context, including system instructions, plans, user turns, retrieved documents, tool outputs, and intermediate reasoning, whose key-value (KV) cache can become a major memory bottleneck.
arXiv:2602. 10238v2 Announce Type: replace-cross Abstract: The growing size of Large Language Models (LLMs) makes efficient inference challenging, primarily due to the memory demands of the autoregressive Key-Value (KV) cache.
arXiv:2608. 05326v1 Announce Type: new Abstract: Autoregressive large language model inference is increasingly constrained by the memory footprint of the Key-Value (KV) cache.
arXiv:2610.06927v1 Announce Type: cross Abstract: The key-value (KV) cache of autoregressive transformers grows linearly with context length and dominates memory at long context. Most training-free r...
The paper proposes LaProx, a new KV cache eviction strategy for long‑context LLM inference that redefines the problem as an output‑aware, layer‑wise matrix multiplication approximation. Unlike prior methods that rely only on local attention weights, LaProx explicitly models the interaction between attention maps and projected value states, accounting for inter‑head dependencies to assign globally comparable token importance scores. Experiments on 19 datasets, including LongBench and Needle‑In‑A‑Haystack, show that LaProx retains model performance while using only 5% of the KV cache and reduces accuracy loss by up to 2× under extreme compression compared to state‑of‑the‑art baselines.
DeferKV rethinks when to evict key‑value (KV) cache entries in long‑context large language models. By delaying eviction until the first decoding step and combining prompt‑side and decode‑side attention signals, it aligns KV importance estimation with actual generation needs. The method requires no extra training or modules and consistently improves performance on benchmarks while keeping latency low.
arXiv:2607. 27600v1 Announce Type: new Abstract: Key-value (KV) cache management through compression and eviction strategies has emerged as an important research direction in recent years.
arXiv:2607. 16213v1 Announce Type: new Abstract: Large Language Models (LLMs) generate text autoregressively, relying on a key-value (KV) cache whose memory footprint grows linearly with context length, creating a major bottleneck.
arXiv:2606. 01563v1 Announce Type: new Abstract: Autoregressive decoding in Transformer-based language models relies on the KV cache, whose memory footprint grows linearly with sequence length and becomes the primary bottleneck for long-context inference.
arXiv:2608.21362v1 Announce Type: new Abstract: Transformer-based large language models (LLMs) incur high prefill latency because key-value (KV) tensors must be recomputed for each request. Existing...
arXiv:2607. 20458v1 Announce Type: cross Abstract: Large language model (LLM) agents operating over extended dialogues accumulate vast amounts of information, yet existing memory systems either retain everything indiscriminately or apply uniform forgetting heuristics that fail to distinguish relevant from irrelevant knowledge.
arXiv:2605. 25475v2 Announce Type: replace-cross Abstract: Large Language Models (LLMs) are increasingly expected to operate over long contexts, yet standard softmax attention incurs a KV cache that grows linearly with sequence length, quickly becoming the bottleneck for long context inference.