The paper introduces D-Quant, a KV cache quantization framework that addresses the memory bottleneck of large language models by using a drift mechanism to convert entropy-coded representations into fixed-size bitstreams. This approach leverages the non-uniform distribution of KV cache values—after rotation and normalization, they approximate a normal distribution—allowing entropy coding to assign shorter codewords to frequent symbols while maintaining regular memory layouts suitable for parallel attention kernels. D-Quant thus aims to reduce memory footprint and bandwidth usage without sacrificing performance.
By Yi Su, Hong Liu, Guanghua Yu, Jianchen Zhu
arXiv:2607. 07144v1 Announce Type: new Abstract: The key-value (KV) cache dominates the memory cost of long-context autoregressive inference, and a growing body of work compresses it through quantization, eviction, or offloading.
By Vladimir Gusev
arXiv:2608. 02901v1 Announce Type: new Abstract: The key-value (KV) cache is the primary memory bottleneck in long-context LLM inference.
By Malik Khalaf, Yara Shamshoum, Nitzan Hodos, Yuval Sieradzki, Assaf Schuster
arXiv:2610.15384v1 Announce Type: cross
Abstract: Compressing the key-value (KV) cache is essential for serving long-context large language models (LLMs) under memory constraints. While current scala...
By Sasho Nedelkoski, Alexander Acker, Soeren Becker, Dominik Scheinert, Odej Kao
VFold introduces a symmetry‑aware value cache merging strategy that compresses the key‑value cache used during Large Language Model decoding. By exploiting inter‑layer cache similarities, it reduces memory usage without causing performance loss or requiring architectural changes. The method can be combined with other compression techniques, such as high‑ratio quantization or key cache pruning, to achieve higher compression ratios with minimal extra cost.
arXiv:2605.19660v3 Announce Type: replace
Abstract: The rapid advancement toward long-context reasoning and multi-modal intelligence has made the memory footprint of the Key-Value (KV) cache a domina...
By Zunhai Su, Rui Yang, Chao Zhang, Yaxiu Liu, Yifan Zhang, Wei Wu, Jing Xiong, Dayou Du, Xialie Zhuang, Yulei Qian, Jianchao Tan, Yuchen Xie, Yik-Chung Wu, Hongxia Yang, Ngai Wong
arXiv:2505. 18231v3 Announce Type: replace-cross Abstract: Large Language Model (LLM) inference is typically memory-intensive, especially when processing large batch sizes and long sequences, due to the large size of key-value (KV) cache.
By Donghyun Son, Euntae Choi, Sungjoo Yoo
arXiv:2410. 13056v4 Announce Type: replace-cross Abstract: Large Language Models (LLMs) have demonstrated remarkable success across a wide range of language tasks, but their deployment on edge devices remains challenging due to the substantial memory requirements imposed by their large parameter sizes.
By Zihan Chen, Bike Xie, Jundong Li, Cong Shen
MultiAlloc is a profiling‑guided framework that allocates limited quantization resources between layer‑wise precision enhancement and token‑wise preservation for KV cache quantization in large language models. By using lightweight offline profiling to assess layer sensitivity and token importance, it automatically derives an effective precision allocation strategy that satisfies a fixed memory budget without online search. Experiments on reasoning and long‑context benchmarks show that MultiAlloc consistently outperforms existing layer‑wise, token‑wise, and multi‑granularity quantization methods across models such as Mistral, Llama, DeepSeek‑Qwen, and QwQ.
PN-Mix is a mixed‑precision key‑cache quantization technique for RoPE‑based decoder‑only large language models that allocates higher precision to salient channel pairs identified during the prefill stage. By focusing on channel pairs with large query–key pair‑norm products, PN‑Mix reduces logit variation caused by quantization errors. Experiments on reasoning, code‑generation, and AIME benchmarks show that PN‑Mix achieves comparable or better accuracy than prior methods while using an average precision of 2.26 bits and degrades gracefully down to 1.5 bits.
By Gil Jae Song, Sungjoo Yoo
arXiv:2609.38121v1 Announce Type: new
Abstract: KV cache memory and bandwidth costs grow with context length and batch size, which limits efficient long-context inference. To address this bottleneck,...
By Jiale Chen, Vage Egiazarian, Eldar Kurti\'c, Torsten Hoefler, Dan Alistarh
arXiv:2608. 14191v1 Announce Type: new Abstract: The key-value (KV) cache stores information from past tokens and is a major memory bottleneck in long-context inference.
By Hannah Laus, Claudio Mayrink Verdun, Hao Wang, Flavio du Pin Calmon, Felix Krahmer