arXiv:2606. 24033v1 Announce Type: new Abstract: Existing low-bit KV-cache quantizers often treat each cached key as a flat vector.
arXiv:2606. 20474v1 Announce Type: cross Abstract: Context-heavy agents place unusual pressure on the key-value (KV) cache: long prefixes are reused across many short turns, while concurrency determines whether the serving system can keep GPUs utilized.
arXiv:2609.05764v1 Announce Type: cross Abstract: The key-value (KV) cache is the dominant memory bottleneck in long-context large language model (LLM) decoding: every step reads it entirely, so deco...
arXiv:2605. 06675v2 Announce Type: replace Abstract: Large language models cache all previously computed key-value (KV) pairs during generation, and this KV cache grows linearly with sequence length, making it a primary memory bottleneck for serving.
Minima-KV introduces a retention‑preserving hierarchy for mixed‑format paged attention that keeps recent and protected anchor pages in FP8 while older pages are compressed into packed TQ3, allowing every live‑request page to remain addressable. The approach uses format‑specific kernels and a globally normalized online‑softmax merge to compute partial attention states, enabling direct heterogeneous decoding without a dense shadow cache. Experiments on Qwen3.6‑27B on a 96‑GB NVIDIA RTX PRO 6000 Blackwell GPU show 3.50× compression over BF16 and 1.75× over FP8, with minimal impact on performance across long‑context benchmarks.
arXiv:2608. 08081v1 Announce Type: cross Abstract: Large mixture-of-experts (MoE) language models with 26--120 billion parameters exceed the memory capacity of consumer devices through three simultaneous pressures: resident weight matrices, key-value (KV) cache state that grows linearly with context, and dozens of expert sublayers that must be paged on demand.
The paper investigates how to recover language model quality lost when using low‑bit key–value caches for autoregressive decoding. By keeping the quantizer fixed and distilling the full‑precision cache behavior into low‑rank Q/K/V projection updates, the authors demonstrate that 4‑bit affine‑cache adapters recover roughly 54 % of the perplexity gap on TinyLlama‑1.1B and 76 % on Gemma‑4‑12B, while preserving most long‑context retrieval. Even a 2‑bit rank–token sweep can dramatically reduce TinyLlama’s perplexity, though it only partially restores retrieval performance.
arXiv:2609.24298v1 Announce Type: new Abstract: What limits KV-cache compression at extreme bit-rates? We argue that it is not the choice of compression scheme, but how its budget is allocated across...
The paper introduces D-Quant, a KV cache quantization framework that addresses the memory bottleneck of large language models by using a drift mechanism to convert entropy-coded representations into fixed-size bitstreams. This approach leverages the non-uniform distribution of KV cache values—after rotation and normalization, they approximate a normal distribution—allowing entropy coding to assign shorter codewords to frequent symbols while maintaining regular memory layouts suitable for parallel attention kernels. D-Quant thus aims to reduce memory footprint and bandwidth usage without sacrificing performance.
arXiv:2609.36760v1 Announce Type: new Abstract: Multi-Head Latent Attention (MLA) enables expressive multi-head attention with compact caches for its content and decoupled RoPE paths, yet cache memor...
arXiv:2607. 01065v1 Announce Type: new Abstract: The deployment of Large Language Models (LLMs) with extended context windows is increasingly constrained by the linear growth of Key-Value (KV) cache memory.
arXiv:2608.30564v1 Announce Type: cross Abstract: Mixed-precision quantization (MPQ) assigns a different bitwidth to each linear layer of a large language model (LLM) to minimize the quantization-ind...
arXiv:2608. 07915v1 Announce Type: new Abstract: Large language models (LLMs) increasingly read long inputs in the agentic era, from whole documents and codebases to conversations across many turns.