The paper introduces ReadKV, a query‑adaptive quantization method for KV caches that stores keys and values in progressive codes, allowing each decoding query to allocate different precision prefixes for keys and values. By optimizing a calibrated distortion objective under fixed budgets, ReadKV achieves near‑optimal allocation and demonstrates that query‑dependent access can outperform any query‑independent reader at the same read budget. Experiments on six base models show that reading only four bits on average from an eight‑bit cache increases perplexity by at most 0.66% while reducing logical reads and retained capacity compared to a 16‑bit cache, and it achieves lower latency than existing codecs on an NVIDIA A10G.
By Siddharth Bhandari, Lucas Gretta, Krishna Balasubramanian, Shiva Kasiviswanathan
The paper introduces a multi‑shell decoder for Leech‑lattice vector quantization, achieving the best reported 2‑bit quality under its evaluation protocol. It presents a GPU‑friendly layout that fuses dequantization with matrix‑vector multiplication, demonstrating significant speed and memory advantages over traditional one‑hot masks and other 4‑bit methods. Experiments show the new kernel outperforms baseline approaches across multiple model sizes, with measurable gains in throughput and reduced byte traffic.
By Pier-Jean Malandrino (Scub)
arXiv:2608. 07915v1 Announce Type: new Abstract: Large language models (LLMs) increasingly read long inputs in the agentic era, from whole documents and codebases to conversations across many turns.
By Jiamu Zhang, Liang Wu, Kelly Wan, Hanjie Chen, Liangjie Hong
arXiv:2607. 07144v1 Announce Type: new Abstract: The key-value (KV) cache dominates the memory cost of long-context autoregressive inference, and a growing body of work compresses it through quantization, eviction, or offloading.
By Vladimir Gusev
arXiv:2606. 20474v1 Announce Type: cross Abstract: Context-heavy agents place unusual pressure on the key-value (KV) cache: long prefixes are reused across many short turns, while concurrency determines whether the serving system can keep GPUs utilized.
By Inesh Chakrabarti (Advanced Micro Devices, University of California, Los Angeles), David Limpus (Advanced Micro Devices, Purdue University), Aditi Ghai Rana (Advanced Micro Devices), Bowen Bao (Advanced Micro Devices), Spandan Tiwari (Advanced Micro Devices), Thiago Crepaldi (Advanced Micro Devices), Ashish Sirasao (Advanced Micro Devices)
BellowsKV introduces a training‑free approach that maps the hidden states of multiple adjacent layers in a large language model to a single shared latent representation per token, allowing each layer to retrieve its own keys and values via a layer‑specific projection. The method retains the latent at full dimension and offers two compression strategies: BellowsKV‑Quant, which stores the latent at low precision, and BellowsKV‑Evict, which makes a shared eviction decision per block. Experiments on Llama‑3.1‑8B, Mistral‑7B, and Qwen2.5‑7B show that BellowsKV‑Quant preserves about 99% of full‑cache accuracy at 3.6× compression and 97‑98% at 13‑14×, outperforming existing cross‑layer methods that drop to at most 84% accuracy at 16× compression.
By Jiamu Zhang, Hanjie Chen
arXiv:2606. 21633v2 Announce Type: replace Abstract: The KV cache dominates GPU memory in long-context LLM serving, crowding out batch capacity and leaving GPU compute idle.
By Omin Kwon, Doyeon Kim, Jongseok Park, Seung Yul Lee, Ion Stoica, Jae W. Lee
arXiv:2609.03949v2 Announce Type: replace-cross
Abstract: A long-lived KV cache must be compressed before the queries that will read it exist. Selection by observed attention collapses there: on a No...
By WenJie Fan
The paper introduces GraphTok, a GPU-based tokenizer that executes the entire tokenization pipeline as a single CUDA graph, matching reference token IDs for models such as GPT‑2, Llama 3, Qwen 2.5, DeepSeek‑V3, and Gemma 3. GraphTok achieves significant speedups, encoding one million GPT‑2 tokens in 1.7–3.2 ms and outperforming the leading CPU tokenizer gigatoken by 4.7–6.0× on Llama 3. In a vLLM deployment with cached prefixes, it reduces mean time‑to‑first‑token by 2.1–2.4× and dramatically increases request throughput, sustaining 518 req/s of 121k‑token traffic versus 296 with gigatoken.
By Mohammad Siavashi, Pascal Schwinte, Ali Banaei, Gerald Q. Maguire Jr., Marco Chiesa, Dejan Kosti\'c
Minima-KV introduces a retention‑preserving hierarchy for mixed‑format paged attention that keeps recent and protected anchor pages in FP8 while older pages are compressed into packed TQ3, allowing every live‑request page to remain addressable. The approach uses format‑specific kernels and a globally normalized online‑softmax merge to compute partial attention states, enabling direct heterogeneous decoding without a dense shadow cache. Experiments on Qwen3.6‑27B on a 96‑GB NVIDIA RTX PRO 6000 Blackwell GPU show 3.50× compression over BF16 and 1.75× over FP8, with minimal impact on performance across long‑context benchmarks.
By Sergii Kozyrev (Minima AI, Inc), Davyd Maiboroda (Minima AI, Inc)
arXiv:2606. 24033v1 Announce Type: new Abstract: Existing low-bit KV-cache quantizers often treat each cached key as a flat vector.
By Fengfeng Liang, Yuechen Zhang, Jiaya Jia
arXiv:2606. 09916v1 Announce Type: cross Abstract: Multi-turn LLM agents fan short queries into long trajectories of tool calls, search results, and intermediate reasoning.
By Junjie Li, Jiong Lou, Jie Li