Large language models (LLMs) exhibit exceptional general language processing capabilities, but their memory and compute costs hinder deployment. Ternarization has emerged as a promising compression technique, offering significant reductions in model size and inference complexity.
arXiv:2606. 13054v1 Announce Type: cross Abstract: Large language models (LLMs) exhibit exceptional general language processing capabilities, but their memory and compute costs hinder deployment.
By Zhixiong Zhao, Zukang Xu, Zhixuan Chen, Xing Hu, Zhe Jiang, Dawei Yang
The paper introduces a unified Hessian-guided framework for post‑training quantization of large language models. By deriving a Hessian‑based approximation of output changes, it jointly optimizes quantization ranges, linear transformations, and weight rounding under a single output‑sensitivity objective. The method remains practical for modern models and consistently outperforms existing transformation and rounding techniques in both channel‑wise and block‑wise settings.
By Mads Toftrup, Marco Federici, Boris van Breugel, Paul Whatmough, Markus Nagel
The paper examines weight-only post‑training quantization (PTQ) and finds that minimizing reconstruction loss does not always improve model performance, sometimes even harming it on calibration data. It shows that weights with low reconstruction loss on one distribution can perform worse when activation distributions shift. To address this, the authors introduce Distributionally Robust Quantization (DRQ), a refinement step that minimizes worst‑case reconstruction loss over a constrained set of activation distributions while keeping the quantization grid unchanged. Experiments demonstrate that DRQ improves models quantized by several PTQ methods across dense and mixture‑of‑experts large language models, offering better downstream performance without extra inference cost.
By Yanlong Zhao, Xiaoyuan Cheng, Huihang Liu, Baihua He, Xinyu Zhang, Harrison Bo Hua Zhu, Wenlong Chen, Li Zeng, Zhuo Sun
The paper presents a comprehensive evaluation of post‑training quantization (PTQ) techniques for large language models (LLMs), addressing gaps in prior work by conducting fair, large‑scale experiments on text‑only LLMs. It decouples quantization into pre‑quantization transformation and error mitigation, systematically studies granularity, symmetry, and data formats (MXFP4, NVFP4), and finds that rotation‑based transformations and low‑rank compensation with GPTQ can improve performance, while finer granularity and asymmetric quantization benefit activations more than weights. The study highlights the importance of understanding component interactions and the impact of scaling factor formats on low‑bit quantization performance.
By Yutong Liu, Cairong Zhao, Guosheng Hu
The paper introduces a unified Hessian-guided framework for post-training quantization of large language models. It derives a Hessian-based approximation of output changes due to joint weight and activation quantization, enabling simultaneous optimization of quantization ranges, linear transformations, and weight rounding under a single output-sensitivity objective. The method remains practical for modern models and consistently outperforms existing transformation and rounding techniques in both channel-wise and block-wise settings.