Hugging Face Blog
Unlocking Longer Generation with Key-Value Cache Quantization
Read the original on Hugging Face Blog →The Flow has not summarised this story yet — read it at Hugging Face Blog.
Related stories
arXiv Computation and Language
Sep 18
D-Quant: Driftable Entropy Coding for KV Cache Quantization
The paper introduces D-Quant, a KV cache quantization framework that addresses the memory bottleneck of large language models by using a drift mechanism to convert entropy-coded representations into fixed-size bitstreams. This approach leverages the non-uniform distribution of KV cache values—after rotation and normalization, they approximate a normal distribution—allowing entropy coding to assign shorter codewords to frequent symbols while maintaining regular memory layouts suitable for parallel attention kernels. D-Quant thus aims to reduce memory footprint and bandwidth usage without sacrificing performance.
arXiv Machine Learning
Jul 9
Fractal KV-Cache Archives: Lossless Symbolic Storage with In-Place Retrieval for Long-Context LLM Inference
arXiv:2607. 07144v1 Announce Type: new Abstract: The key-value (KV) cache dominates the memory cost of long-context autoregressive inference, and a growing body of work compresses it through quantization, eviction, or offloading.
arXiv Machine Learning
Aug 5
AnchorKV: Anchor-Residual KV Cache Compression
arXiv:2608. 02901v1 Announce Type: new Abstract: The key-value (KV) cache is the primary memory bottleneck in long-context LLM inference.
arXiv AI
Oct 15
E8 Lattice Key Quantization for Low-Bit KV-Cache Compression
arXiv:2610.15384v1 Announce Type: cross Abstract: Compressing the key-value (KV) cache is essential for serving long-context large language models (LLMs) under memory constraints. While current scala...
Hugging Face Trending Papers
Oct 8
VFold: Symmetry-Aware Cross-Layer Value Cache Compression
VFold introduces a symmetry‑aware value cache merging strategy that compresses the key‑value cache used during Large Language Model decoding. By exploiting inter‑layer cache similarities, it reduces memory usage without causing performance loss or requiring architectural changes. The method can be combined with other compression techniques, such as high‑ratio quantization or key cache pruning, to achieve higher compression ratios with minimal extra cost.
arXiv Machine Learning
Oct 15
OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond
arXiv:2605.19660v3 Announce Type: replace Abstract: The rapid advancement toward long-context reasoning and multi-modal intelligence has made the memory footprint of the Key-Value (KV) cache a domina...