arXiv AI
Aug 6
Spend Bits Where Queries Look: KV Cache Vector Quantization with Attention-Preserving Transforms
arXiv:2608. 04074v1 Announce Type: cross Abstract: Long-context LLM decoding reads the key-value (KV) cache at every step.
GSRQ: Gain-Shape Residual Quantization for Sub-1-bit KV Cache
arXiv:2607. 01065v1 Announce Type: new Abstract: The deployment of Large Language Models (LLMs) with extended context windows is increasingly constrained by the linear growth of Key-Value (KV) cache memory.
Related stories
arXiv AI
Jul 16
NSNQuant: A Double Normalization Approach for Calibration-Free Low-Bit Vector Quantization of KV Cache
arXiv:2505. 18231v3 Announce Type: replace-cross Abstract: Large Language Model (LLM) inference is typically memory-intensive, especially when processing large batch sizes and long sequences, due to the large size of key-value (KV) cache.
arXiv Machine Learning
Jun 3
KVarN: Variance-Normalized KV-Cache Quantization Mitigates Error Accumulation in Reasoning Tasks
arXiv:2606. 03458v1 Announce Type: new Abstract: Test-time scaling is a powerful approach to obtain better reasoning in large language models, but it becomes memory-bottlenecked during long-horizon decoding, as the KV-cache grows.
arXiv Machine Learning
Jul 9
Fractal KV-Cache Archives: Lossless Symbolic Storage with In-Place Retrieval for Long-Context LLM Inference
arXiv:2607. 07144v1 Announce Type: new Abstract: The key-value (KV) cache dominates the memory cost of long-context autoregressive inference, and a growing body of work compresses it through quantization, eviction, or offloading.
arXiv Machine Learning
Jun 29
RateQuant: Optimal Mixed-Precision KV Cache Quantization via Rate-Distortion Theory
arXiv:2605. 06675v2 Announce Type: replace Abstract: Large language models cache all previously computed key-value (KV) pairs during generation, and this KV cache grows linearly with sequence length, making it a primary memory bottleneck for serving.
arXiv Machine Learning
Jun 8
AAAC: Activation-Aware Adaptive Codebooks for 4-bit LLM Weight Quantization
arXiv:2605. 08692v2 Announce Type: replace Abstract: Post-training weight-only quantization to 4 bits is widely used to reduce the memory and compute costs of large language model inference.
arXiv Machine Learning
Jun 24
RoPE-Aware Bit Allocation for KV-Cache Quantization
arXiv:2606. 24033v1 Announce Type: new Abstract: Existing low-bit KV-cache quantizers often treat each cached key as a flat vector.
arXiv AI
Aug 5
Output-Aware Rotation for INT2 KV-Cache Quantization
arXiv:2608. 02691v1 Announce Type: cross Abstract: The key-value (KV) cache has become a major memory and bandwidth bottleneck in long-context large language model inference, making ultra-low-bit quantization increasingly important.
arXiv Machine Learning
Jul 7
Variable Bit-width Quantization: Learning Per-Group Precision for "Bigger-but-Smaller" Language Models
arXiv:2607. 02893v1 Announce Type: new Abstract: Low-bit quantization shrinks language models but treats precision as a single global hyper-parameter: every weight uses the same bit-width.
arXiv AI
Jul 21
High-accuracy Low-Bit KV-Cache Quantization via Local Distribution Restoration
arXiv:2607. 16248v1 Announce Type: cross Abstract: Long-context large language model inference relies on the KV cache to avoid redundant attention computation, but incurs high memory and bandwidth overheads.
arXiv Machine Learning
Jul 15
A JoLT for the KV Cache: Near-Lossless KV Cache Compression via Joint Tucker and JL-Residual Allocation for LLMs
arXiv:2607. 12550v1 Announce Type: new Abstract: The key-value (KV) cache has become the dominant memory cost of transformer inference.
arXiv Machine Learning
2d ago
KV Cache Compression Through the Lens of Transform Coding
arXiv:2608. 14191v1 Announce Type: new Abstract: The key-value (KV) cache stores information from past tokens and is a major memory bottleneck in long-context inference.