The paper introduces PASK (Parser-Aware Structural KV Persistence), a method that leverages parser transitions to inform key‑value (KV) persistence decisions in structured generation tasks. By aligning KV compression with task‑level structured risk, PASK sets protection floors based on error sensitivity and allocates remaining KV capacity using attention‑output distortion, producing a lightweight, structure‑conditioned lookup policy. In experiments on Qwen3‑4B, PASK achieves a 17.39‑point accuracy gain over the best compressed baseline, while delivering up to 2.2× higher throughput, 3.3× lower TPOT, and 0.53× the peak GPU memory of full KV.
arXiv:2606. 17872v1 Announce Type: cross Abstract: Large language models (LLMs) outperform earlier architectures on generative inference and long-context tasks, but their large size introduces significant challenges in memory usage, energy cost, and on-device deployment.
The paper introduces DCKV, a differentiable codec designed to compress the key-value (KV) cache used in hybrid language models. DCKV jointly optimizes its transform, quantizer, and entropy models—conditioned on prompt tokens and neighboring positions—directly against a frozen language model, balancing compression rate with minimal impact on output quality. Experiments on Qwen3.5-2B demonstrate that DCKV can reduce the storage requirement for a compressed token by over 55× compared to bf16, with a loss of no more than 1.8 points on long‑context benchmarks, and by over 100× with a loss of no more than 3.6 points.
DeferKV rethinks when to evict key‑value (KV) cache entries in long‑context large language models. By delaying eviction until the first decoding step and combining prompt‑side and decode‑side attention signals, it aligns KV importance estimation with actual generation needs. The method requires no extra training or modules and consistently improves performance on benchmarks while keeping latency low.
arXiv:2610.08811v1 Announce Type: new Abstract: As context windows scale to tens or hundreds of thousands of tokens, KV cache compression has become essential for efficient LLM inference. Existing me...
arXiv:2608.21362v1 Announce Type: new Abstract: Transformer-based large language models (LLMs) incur high prefill latency because key-value (KV) tensors must be recomputed for each request. Existing...
arXiv:2607. 08057v1 Announce Type: cross Abstract: Despite the rapid advancements of large language models (LLMs), LLM serving systems remain memory-intensive and costly.
arXiv:2608. 15797v1 Announce Type: new Abstract: KV-cache eviction caps the memory cost of long reasoning traces but is inherently lossy because the model decodes from a partial view of its history.
arXiv:2606. 06256v1 Announce Type: new Abstract: As the input length of large language model (LLM) serving continues to grow, the KV cache has become a dominant bottleneck in AI infrastructure.
arXiv:2608. 01651v1 Announce Type: cross Abstract: Hybrid-attention large language models combine full attention with recurrent linear attention to reduce long-context inference costs, yet their autoregressive decoding remains memory-bound.
arXiv:2602. 08686v3 Announce Type: replace-cross Abstract: Prefill-only KV compression freezes a token subset at the end of prefill and decodes from it without further eviction.
arXiv:2609.07966v1 Announce Type: new Abstract: Key--value (KV) cache compression is an effective way to reduce the memory overhead of large language model (LLM) inference, particularly for long-cont...
arXiv:2607. 05147v1 Announce Type: new Abstract: Speculative decoding accelerates Large Language Model (LLM) inference by decoupling draft generation from target verification.