arXiv Computation and Language September 18, 2026 By Yi Su, Hong Liu, Guanghua Yu, Jianchen Zhu

D-Quant: Driftable Entropy Coding for KV Cache Quantization

Read the original on arXiv Computation and Language →

The paper introduces D-Quant, a KV cache quantization framework that addresses the memory bottleneck of large language models by using a drift mechanism to convert entropy-coded representations into fixed-size bitstreams. This approach leverages the non-uniform distribution of KV cache values—after rotation and normalization, they approximate a normal distribution—allowing entropy coding to assign shorter codewords to frequent symbols while maintaining regular memory layouts suitable for parallel attention kernels. D-Quant thus aims to reduce memory footprint and bandwidth usage without sacrificing performance.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

arXiv AI

Aug 6

Spend Bits Where Queries Look: KV Cache Vector Quantization with Attention-Preserving Transforms

arXiv:2608. 04074v1 Announce Type: cross Abstract: Long-context LLM decoding reads the key-value (KV) cache at every step.

By Samuel Fern\'andez-Mendui\~na, Amir Ziashahabi, Eduardo Pavez, Antonio Ortega, Salman Avestimehr

llms efficiency

arXiv AI

Jul 16

NSNQuant: A Double Normalization Approach for Calibration-Free Low-Bit Vector Quantization of KV Cache

arXiv:2505. 18231v3 Announce Type: replace-cross Abstract: Large Language Model (LLM) inference is typically memory-intensive, especially when processing large batch sizes and long sequences, due to the large size of key-value (KV) cache.

By Donghyun Son, Euntae Choi, Sungjoo Yoo

llms efficiency safety

arXiv Machine Learning

Jul 9

Fractal KV-Cache Archives: Lossless Symbolic Storage with In-Place Retrieval for Long-Context LLM Inference

arXiv:2607. 07144v1 Announce Type: new Abstract: The key-value (KV) cache dominates the memory cost of long-context autoregressive inference, and a growing body of work compresses it through quantization, eviction, or offloading.

By Vladimir Gusev

llms efficiency

arXiv Machine Learning

Jul 2

GSRQ: Gain-Shape Residual Quantization for Sub-1-bit KV Cache

arXiv:2607. 01065v1 Announce Type: new Abstract: The deployment of Large Language Models (LLMs) with extended context windows is increasingly constrained by the linear growth of Key-Value (KV) cache memory.

By Soosung Kim, Minjae Park, Eui-Young Chung, Jaeyong Chung

llms efficiency safety

arXiv Machine Learning

Jun 29

RateQuant: Optimal Mixed-Precision KV Cache Quantization via Rate-Distortion Theory

arXiv:2605. 06675v2 Announce Type: replace Abstract: Large language models cache all previously computed key-value (KV) pairs during generation, and this KV cache grows linearly with sequence length, making it a primary memory bottleneck for serving.

By Fei Zuo, Zikang Zhou, Hao Cong, Xiaoyan Xi, Ho Fai Leung

llms efficiency

arXiv Machine Learning

Sep 16

LLM Inference in a Flash!

The paper "LLM Inference in a Flash!" proposes an integer‑only quantization scheme and a dictionary‑based KV cache compression technique to enable large language model inference on compute‑in‑flash (CIF) devices. By eliminating floating‑point operations and reducing KV cache traffic through sparse dictionary coding, the authors achieve minimal accuracy loss while cutting dynamic KV cache traffic by 15× on Llama‑3.1‑8B and Qwen‑2.5‑7B models.

By Sebastian Zhao, Minseo Kim, Coleman Hooper, Luca Manolache, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, Amir Gholami

llms rag nlp efficiency

One email a morning, machine-written

Related stories

Spend Bits Where Queries Look: KV Cache Vector Quantization with Attention-Preserving Transforms

NSNQuant: A Double Normalization Approach for Calibration-Free Low-Bit Vector Quantization of KV Cache

Fractal KV-Cache Archives: Lossless Symbolic Storage with In-Place Retrieval for Long-Context LLM Inference

GSRQ: Gain-Shape Residual Quantization for Sub-1-bit KV Cache

RateQuant: Optimal Mixed-Precision KV Cache Quantization via Rate-Distortion Theory

LLM Inference in a Flash!