arXiv Machine Learning

ResidualQuant: KV Cache Quantization for Looped Transformers with 2-Bit Residuals

ResidualQuant introduces a 2‑bit residual quantization scheme for KV caches in looped Transformers, leveraging the similarity of KV states across loops. By using the final‑loop KV states as a reference and applying least‑square scaling, rotations, and loop‑wise mixed precision, the method achieves accurate quantization down to INT2. Experiments on various looped Transformer models show that ResidualQuant improves the accuracy‑memory tradeoff, retaining near BF16 accuracy while reducing KV storage by 80.7% and boosting decode throughput and batch size on an RTX 5090.

arXiv Machine Learning

Sep 25

FlashLoop: Fast and Memory-Efficient Looped Transformers via Lazy Updates

FlashLoop is a training‑free inference framework for Looped Transformers that reduces cross‑loop redundancy by employing token‑sparse updates, sparse attention, and KV‑residual quantization. It exploits observations that, as loops progress, state changes concentrate on a small token subset, attention differences are dominated by a sparse key subset, and KV residuals become amenable to low‑bit quantization. The method achieves lossless accuracy with up to 1.64× speedup and 6× KV‑cache memory reduction across several Looped Transformer models.

By Wanqi Yang, Shiwei Liu

llms efficiency

arXiv AI

2d ago

LoopQuant: Accurate and Efficient Post-Training Quantization for Looped Transformers

arXiv:2610.16368v1 Announce Type: cross Abstract: Looped Language Models (LoopLMs) improve parameter efficiency by recursively reusing shared Transformer modules, yet their repeated computation over...

By Zhixiong Zhao, Zukang Xu, Xing Hu, Lifeng Liu, Guangyu Sun, Dawei Yang

llms efficiency

arXiv Machine Learning

Sep 30

WUSH-KV: KV Cache Quantization with Data-Adaptive Transforms

arXiv:2609.38121v1 Announce Type: new Abstract: KV cache memory and bandwidth costs grow with context length and batch size, which limits efficient long-context inference. To address this bottleneck,...

By Jiale Chen, Vage Egiazarian, Eldar Kurti\'c, Torsten Hoefler, Dan Alistarh

efficiency

arXiv AI

Jul 7

ARCQuant: Boosting NVFP4 Quantization with Augmented Residual Channels for LLMs

arXiv:2601. 07475v2 Announce Type: replace-cross Abstract: The emergence of fine-grained numerical formats like NVFP4 presents new opportunities for efficient Large Language Model (LLM) inference.

By Haoqian Meng, Yilun Luo, Yafei Zhao, Wenyuan Liu, Peng Zhang, Xindian Ma

llms efficiency benchmarks

arXiv Machine Learning

Sep 22

PRQuant: Permutation Residual Quantization for Low-Overhead Inference

PRQuant introduces a training‑free, low‑overhead method for low‑bit quantization of linear layers by permuting input channels that cause the largest quantization error into contiguous tail blocks and precomputing residual weight sub‑tensors. The approach eliminates the need for online gathering during inference, converting scattered residual compensation into a regular tail‑augmented GEMM and thereby reducing latency. Experiments show that PRQuant lowers down‑projection reconstruction error and outperforms standard MXFP4 and other post‑training quantization baselines on five downstream benchmarks, improving accuracy by up to 1.24 points on Qwen3‑4B‑Instruct‑2507.

By Peiran Wang, Anqi Wang, Jiaying Zhao, Huiwen Yang, Zhenyu Ming, Rongqian Wang, Yiwu Yao, Kun Tian, Xin Yao, Gong Zhang, Fan Yang, Zhongyi Huang

efficiency benchmarks

arXiv AI

Jun 2

GoQuant: Geometric Orthogonal Residual Projection for Multiplier-Free Power-of-Two Transformer Quantization

arXiv:2605. 26092v4 Announce Type: replace-cross Abstract: The deployment of Large Language Models (LLMs) and Vision Transformers (ViTs) on edge devices is significantly constrained by memory limitations and the critical timing bottlenecks introduced by dense Multiply-Accumulate (MAC) arrays.

By Maoyang Xiang, Tao Luo, Bo Wang

llms efficiency

arXiv Machine Learning

Jun 3

KVarN: Variance-Normalized KV-Cache Quantization Mitigates Error Accumulation in Reasoning Tasks

arXiv:2606. 03458v1 Announce Type: new Abstract: Test-time scaling is a powerful approach to obtain better reasoning in large language models, but it becomes memory-bottlenecked during long-horizon decoding, as the KV-cache grows.

By Lorenz K. Muller, Philippe Bich, Chiara Boretti, Hyun-Min Chang, Jiawei Zhuang, Lukas Cavigelli

llms efficiency benchmarks

arXiv AI

Jul 16

NSNQuant: A Double Normalization Approach for Calibration-Free Low-Bit Vector Quantization of KV Cache

arXiv:2505. 18231v3 Announce Type: replace-cross Abstract: Large Language Model (LLM) inference is typically memory-intensive, especially when processing large batch sizes and long sequences, due to the large size of key-value (KV) cache.

By Donghyun Son, Euntae Choi, Sungjoo Yoo

llms efficiency safety

arXiv AI

Oct 5

Tailoring the Quantization Space for 1-Bit KV Cache Compression

arXiv:2610.03027v1 Announce Type: cross Abstract: The key-value (KV) cache becomes a major memory bottleneck in long-context LLM inference, placing substantial pressure on memory capacity and bandwid...

By Minsoo Cheong, Donghyun Son, Sungjoo Yoo

llms efficiency benchmarks

arXiv Machine Learning

2d ago

OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond

arXiv:2605.19660v3 Announce Type: replace Abstract: The rapid advancement toward long-context reasoning and multi-modal intelligence has made the memory footprint of the Key-Value (KV) cache a domina...

By Zunhai Su, Rui Yang, Chao Zhang, Yaxiu Liu, Yifan Zhang, Wei Wu, Jing Xiong, Dayou Du, Xialie Zhuang, Yulei Qian, Jianchao Tan, Yuchen Xie, Yik-Chung Wu, Hongxia Yang, Ngai Wong

llms efficiency multimodal

arXiv AI

Jul 24

Codec-Gauge: Learning Compression-Friendly Gauges for Transformer KV Caches

arXiv:2607. 20538v1 Announce Type: cross Abstract: Long-context Transformer inference increasingly relies on KV-cache compression or quantization.

By Yitao Jiang, Yaoqing Yang, Luyang Zhao, Muhao Chen, Devin Balkcom

llms efficiency

arXiv Machine Learning

Jul 2

GSRQ: Gain-Shape Residual Quantization for Sub-1-bit KV Cache

arXiv:2607. 01065v1 Announce Type: new Abstract: The deployment of Large Language Models (LLMs) with extended context windows is increasingly constrained by the linear growth of Key-Value (KV) cache memory.

By Soosung Kim, Minjae Park, Eui-Young Chung, Jaeyong Chung

llms efficiency safety

Related stories