SQuAT introduces a shape‑gain vector quantization‑aware training (VQ‑QAT) method that separates weight direction and magnitude, using an $E_8$‑derived spherical codebook for directions and low‑bit scalar quantizers for gains. By applying distinct straight‑through estimators (STE) to each component, SQuAT induces tangent‑space updates that act as an implicit spherical (Riemannian) optimization, improving training dynamics over standard STE approaches. A distributed exact nearest‑neighbor assignment scheme further reduces training time by 12.8×, enabling scaling to 10 B tokens and achieving state‑of‑the‑art accuracy on several large language models.
By Sebastian Loeschcke, Ivan Koryakovskiy, Tycho F. A. van der Ouderaa, Riccardo Del Chiaro, Mart van Baalen, Paul N. Whatmough, Markus Nagel
arXiv:2603. 11021v2 Announce Type: replace Abstract: Scalar quantization of large language models (LLMs) is fundamentally limited by information-theoretic bounds.
By Tycho F. A. van der Ouderaa, Mart van Baalen, Paul Whatmough, Markus Nagel
arXiv:2607. 01065v1 Announce Type: new Abstract: The deployment of Large Language Models (LLMs) with extended context windows is increasingly constrained by the linear growth of Key-Value (KV) cache memory.
By Soosung Kim, Minjae Park, Eui-Young Chung, Jaeyong Chung
arXiv:2610.27699v1 Announce Type: new
Abstract: Low-bit quantization reduces the memory footprint of large language models, but preserving accuracy with efficient training remains challenging. Scalar...
By Xinyu Lei, Hong Huang, Jiebin Zhang, Hongyu Li, Jinlong Song, Decheng Wu, Guanghua Yu, Jianchen Zhu, Sujian Li
arXiv:2605.11222v2 Announce Type: replace
Abstract: Quantization is an effective strategy to reduce the storage and computation footprint of large language models (LLMs). Post-training quantization (...
By Ryan Lucas, Mehdi Makni, Xiang Meng, Adam Deng, Rahul Mazumder
arXiv:2610.09212v1 Announce Type: new
Abstract: The best two-bit weight quantizers for large language models, such as QTIP and Proteus, rotate each weight matrix by a random orthogonal transform, whi...
By Guanhua Ding, Zi Wang, Ruichao Li, Jack Liu
arXiv:2606. 10890v1 Announce Type: cross Abstract: Post-training quantization (PTQ) compresses large language models by mapping weights to low-bit representations.
By Juan Amboage, Pablo Monteagudo-Lago, Ian Colbert, Giuseppe Franco, Nicholas Fraser
arXiv:2606. 26650v1 Announce Type: cross Abstract: In this paper, we present CAT-Q, Cost-efficient and Accurate Ternary Quantization, for compressing and accelerating LLMs.
By Shigeng Wang, Chao Li, Yangyuxuan Kang, Jiawei Fan, Anbang Yao
The paper introduces CanonQ, a unified quantization‑aware training framework that separates source canonicalization from task‑aware adaptation to enable extreme low‑bit LLM compression. By applying fixed rotations and energy normalization, CanonQ maps heterogeneous tensors to canonical coordinates, allowing frozen Gaussian‑reference codebooks to be reused across layers and models. Joint training then adapts the network to the coupled errors of weight, activation, and KV cache quantization, achieving significant performance gains—up to 14.28× lower perplexity and 57.9% higher zero‑shot accuracy on LLaMA models, and substantial improvements on instruction‑tuned MobileLLM‑Pro‑1B.
"whyItMatters":"The framework demonstrates that carefully designed canonicalization and joint training can dramatically improve the efficiency and accuracy of low‑bit quantized large language models, offering a practical path to deploy powerful LLMs with reduced computational and memory footprints."
By Kai Yi, Tarek Elgamal, Sruthikesh Surineni, Vignesh Vivekraja, Soumyadeep Ghosh, Steven Li
arXiv:2606. 10531v1 Announce Type: cross Abstract: Quantization-aware training (QAT) is essential for extremely low-bit large language models (LLMs).
By Haoyu Wang, Xingyu Yu, Haiyan Zhao, Fengxiang Wang, Xu Han
InfoQuant is a train‑free method that reshapes activation distributions to be more compatible with low‑bit quantizers. It uses Peak Suppression Orthogonal Transformation (PSOT) and adaptive outlier‑token selection to reduce numerical range while maintaining dispersion, thereby lowering quantization error. Across several large language model families, InfoQuant outperforms existing post‑training quantization and end‑to‑end training baselines, preserving 97% of floating‑point accuracy on average and narrowing the LLaMA‑2 13B performance gap by 42%.
By Ke Li, Dong An, Xiaoling Zang, Can Ye, Liang Xie, Qibo Qiu, Chen Shen, Xiaofei He, Wenxiao Wang
In this paper, we present CAT-Q, Cost-efficient and Accurate Ternary Quantization, for compressing and accelerating LLMs. Unlike existing state-of-the-art ternary quantization methods that rely on data-intensive and costly quantization-aware training to mitigate severe performance degradation, CAT-Q is a simple yet effective post-training quantization scheme that is readily applicable to LLMs with diverse architectures and model sizes.