SQuAT introduces a shape‑gain vector‑quantization‑aware training (VQ‑QAT) method that separates weight direction and magnitude, using an $E_8$‑derived spherical codebook for directions and low‑bit scalar quantizers for gains. The approach employs distinct straight‑through estimators for shape and gain, enabling tangent‑space updates that act as implicit spherical optimization. Distributed exact nearest‑neighbor assignment reduces training time by 12.8×, allowing scaling to 10 B tokens and yielding state‑of‑the‑art accuracy on several large language models.
arXiv:2603. 11021v2 Announce Type: replace Abstract: Scalar quantization of large language models (LLMs) is fundamentally limited by information-theoretic bounds.
By Tycho F. A. van der Ouderaa, Mart van Baalen, Paul Whatmough, Markus Nagel
arXiv:2607. 01065v1 Announce Type: new Abstract: The deployment of Large Language Models (LLMs) with extended context windows is increasingly constrained by the linear growth of Key-Value (KV) cache memory.
By Soosung Kim, Minjae Park, Eui-Young Chung, Jaeyong Chung
arXiv:2605.11222v2 Announce Type: replace
Abstract: Quantization is an effective strategy to reduce the storage and computation footprint of large language models (LLMs). Post-training quantization (...
By Ryan Lucas, Mehdi Makni, Xiang Meng, Adam Deng, Rahul Mazumder
arXiv:2610.27699v1 Announce Type: new
Abstract: Low-bit quantization reduces the memory footprint of large language models, but preserving accuracy with efficient training remains challenging. Scalar...
By Xinyu Lei, Hong Huang, Jiebin Zhang, Hongyu Li, Jinlong Song, Decheng Wu, Guanghua Yu, Jianchen Zhu, Sujian Li
arXiv:2606. 10890v1 Announce Type: cross Abstract: Post-training quantization (PTQ) compresses large language models by mapping weights to low-bit representations.
By Juan Amboage, Pablo Monteagudo-Lago, Ian Colbert, Giuseppe Franco, Nicholas Fraser
arXiv:2606. 26650v1 Announce Type: cross Abstract: In this paper, we present CAT-Q, Cost-efficient and Accurate Ternary Quantization, for compressing and accelerating LLMs.
By Shigeng Wang, Chao Li, Yangyuxuan Kang, Jiawei Fan, Anbang Yao
arXiv:2610.09212v1 Announce Type: new
Abstract: The best two-bit weight quantizers for large language models, such as QTIP and Proteus, rotate each weight matrix by a random orthogonal transform, whi...
By Guanhua Ding, Zi Wang, Ruichao Li, Jack Liu
InfoQuant is a train‑free method that reshapes activation distributions to be more compatible with low‑bit quantizers. It uses Peak Suppression Orthogonal Transformation (PSOT) and adaptive outlier‑token selection to reduce numerical range while maintaining dispersion, thereby lowering quantization error. Across several large language model families, InfoQuant outperforms existing post‑training quantization and end‑to‑end training baselines, preserving 97% of floating‑point accuracy on average and narrowing the LLaMA‑2 13B performance gap by 42%.
By Ke Li, Dong An, Xiaoling Zang, Can Ye, Liang Xie, Qibo Qiu, Chen Shen, Xiaofei He, Wenxiao Wang
The paper introduces CanonQ, a unified quantization‑aware training framework that separates source canonicalization from task‑aware adaptation to enable extreme low‑bit LLM compression. By applying fixed rotations and energy normalization, CanonQ maps heterogeneous tensors to canonical coordinates, allowing frozen Gaussian‑reference codebooks to be reused across layers and models. Joint training then adapts the network to the coupled errors of weight, activation, and KV cache quantization, achieving significant performance gains—up to 14.28× lower perplexity and 57.9% higher zero‑shot accuracy on LLaMA models, and substantial improvements on instruction‑tuned MobileLLM‑Pro‑1B.
"whyItMatters":"The framework demonstrates that carefully designed canonicalization and joint training can dramatically improve the efficiency and accuracy of low‑bit quantized large language models, offering a practical path to deploy powerful LLMs with reduced computational and memory footprints."
By Kai Yi, Tarek Elgamal, Sruthikesh Surineni, Vignesh Vivekraja, Soumyadeep Ghosh, Steven Li
The paper introduces Trainable Vector Quantization (TVQ), a method that uses straight-through estimation to allow gradient descent optimization of vector-quantized large language models. TVQ reformulates VQ with trainable latent weights, enabling arbitrary objective minimization and is applied in a block-wise knowledge distillation framework to balance quality and training efficiency. Experiments show TVQ achieves state‑of‑the‑art low‑bit post‑training quantization, achieving near‑lossless 2‑bit quantization for LLaMA2‑70B with only a 1% accuracy drop on question‑answering benchmarks.
By Anastasiia Prutianova, Alina Bogdanova, Mikhail Shutov, Dmitrii Poimanov, Anatoly Gorkov, Ilia Zharikov
arXiv:2606. 10531v1 Announce Type: cross Abstract: Quantization-aware training (QAT) is essential for extremely low-bit large language models (LLMs).
By Haoyu Wang, Xingyu Yu, Haiyan Zhao, Fengxiang Wang, Xu Han