arXiv AI
Aug 6
Spend Bits Where Queries Look: KV Cache Vector Quantization with Attention-Preserving Transforms
arXiv:2608. 04074v1 Announce Type: cross Abstract: Long-context LLM decoding reads the key-value (KV) cache at every step.
Dual-QK: Sharp Queries and Flat Keys for Prunable 2-bit KV Caches
Dual-QK introduces paired non‑orthogonal transforms for queries and keys to enable sharp queries and flat keys, improving 2‑bit quantization and dynamic channel pruning of KV caches. The method balances key scales for INT2 quantization while concentrating query energy for pruning, and supports channel‑0 protection and bucket‑relative RoPE for long‑context accuracy. Experiments on four models across five generative benchmarks show that Dual‑QK outperforms OSCAR at 40% query‑channel sparsity, achieving 6.8× KV‑cache compression, an 8.3× reduction in KV read volume, and up to 3.75× decoding throughput over unpruned BF16.
Related stories
arXiv AI
Oct 5
Tailoring the Quantization Space for 1-Bit KV Cache Compression
arXiv:2610.03027v1 Announce Type: cross Abstract: The key-value (KV) cache becomes a major memory bottleneck in long-context LLM inference, placing substantial pressure on memory capacity and bandwid...
arXiv Machine Learning
4d ago
Read What Matters: Query-Adaptive Quantization for KV Caches
The paper introduces ReadKV, a query‑adaptive quantization method for KV caches that stores keys and values in progressive codes, allowing each decoding query to allocate different precision prefixes for keys and values. By optimizing a calibrated distortion objective under fixed budgets, ReadKV achieves near‑optimal allocation and demonstrates that query‑dependent access can outperform any query‑independent reader at the same read budget. Experiments on six base models show that reading only four bits on average from an eight‑bit cache increases perplexity by at most 0.66% while reducing logical reads and retained capacity compared to a 16‑bit cache, and it achieves lower latency than existing codecs on an NVIDIA A10G.
arXiv Machine Learning
Jun 24
RoPE-Aware Bit Allocation for KV-Cache Quantization
arXiv:2606. 24033v1 Announce Type: new Abstract: Existing low-bit KV-cache quantizers often treat each cached key as a flat vector.
arXiv Machine Learning
Sep 15
Grouped Value Attention: Efficient KV Caching via On-Demand Key Reconstruction
arXiv:2609.13285v1 Announce Type: cross Abstract: The KV cache is a primary bottleneck for Transformer decoding: its memory footprint and cache-read traffic grow with sequence length. Grouped-query a...
arXiv AI
Aug 5
Output-Aware Rotation for INT2 KV-Cache Quantization
arXiv:2608. 02691v1 Announce Type: cross Abstract: The key-value (KV) cache has become a major memory and bandwidth bottleneck in long-context large language model inference, making ultra-low-bit quantization increasingly important.
arXiv Machine Learning
Jul 2
GSRQ: Gain-Shape Residual Quantization for Sub-1-bit KV Cache
arXiv:2607. 01065v1 Announce Type: new Abstract: The deployment of Large Language Models (LLMs) with extended context windows is increasingly constrained by the linear growth of Key-Value (KV) cache memory.
arXiv Machine Learning
Sep 25
A JoLT for the KV cache: Near-Lossless KV Cache Compression via Joint Rank-bit Allocation
The paper introduces JoLT, a training‑free compressor that jointly allocates rank and precision for key‑value (KV) cache compression in long‑context language models. JoLT treats grouped prefill caches as fourth‑order tensors, applies partial Tucker decomposition along token and feature modes, and uses a rotated low‑bit quantizer for residuals, all governed by a single Lagrangian dual under a global byte constraint. Across five models from four architecture families, JoLT achieves 2–3× compression with less than 0.2% perplexity loss, and near‑lossless retrieval accuracy on LLaMA‑3.1‑8B at 64K context up to 3× compression.
arXiv Machine Learning
Sep 30
WUSH-KV: KV Cache Quantization with Data-Adaptive Transforms
arXiv:2609.38121v1 Announce Type: new Abstract: KV cache memory and bandwidth costs grow with context length and batch size, which limits efficient long-context inference. To address this bottleneck,...
arXiv AI
Aug 25
KVBoost: Chunk-Level Key-Value Cache Reuse with Deviation-Guided Recomputation for Efficient Large Language Model Inference
arXiv:2608.21362v1 Announce Type: new Abstract: Transformer-based large language models (LLMs) incur high prefill latency because key-value (KV) tensors must be recomputed for each request. Existing...
arXiv AI
Jul 16
NSNQuant: A Double Normalization Approach for Calibration-Free Low-Bit Vector Quantization of KV Cache
arXiv:2505. 18231v3 Announce Type: replace-cross Abstract: Large Language Model (LLM) inference is typically memory-intensive, especially when processing large batch sizes and long sequences, due to the large size of key-value (KV) cache.
arXiv Computation and Language
Sep 18
D-Quant: Driftable Entropy Coding for KV Cache Quantization
The paper introduces D-Quant, a KV cache quantization framework that addresses the memory bottleneck of large language models by using a drift mechanism to convert entropy-coded representations into fixed-size bitstreams. This approach leverages the non-uniform distribution of KV cache values—after rotation and normalization, they approximate a normal distribution—allowing entropy coding to assign shorter codewords to frequent symbols while maintaining regular memory layouts suitable for parallel attention kernels. D-Quant thus aims to reduce memory footprint and bandwidth usage without sacrificing performance.