arXiv Machine Learning

Beyond local reconstruction: Geometry-preserving output alignment for binary LLM quantization

The paper introduces a geometry-preserving output alignment framework for 1‑bit post‑training quantization of large language models. It formulates output alignment as a constrained optimization that balances reconstruction quality with token similarity preservation, and proposes a geometry‑aware salient‑mask selection strategy to choose quantization settings that maintain token relations. Experiments on various LLM families and downstream tasks show consistent gains over existing 1‑bit PTQ methods.

arXiv AI
Oct 31

A Unified Hessian-Guided Framework for Post-Training Quantization

The paper introduces a unified Hessian-guided framework for post‑training quantization of large language models. By deriving a Hessian‑based approximation of output changes, it jointly optimizes quantization ranges, linear transformations, and weight rounding under a single output‑sensitivity objective. The method remains practical for modern models and consistently outperforms existing transformation and rounding techniques in both channel‑wise and block‑wise settings.

By Mads Toftrup, Marco Federici, Boris van Breugel, Paul Whatmough, Markus Nagel
arXiv AI
Oct 9

When Lower Reconstruction Loss Hurts: Distributionally Robust Refinement for Low-Bit LLM Quantization

The paper examines weight-only post‑training quantization (PTQ) and finds that minimizing reconstruction loss does not always improve model performance, sometimes even harming it on calibration data. It shows that weights with low reconstruction loss on one distribution can perform worse when activation distributions shift. To address this, the authors introduce Distributionally Robust Quantization (DRQ), a refinement step that minimizes worst‑case reconstruction loss over a constrained set of activation distributions while keeping the quantization grid unchanged. Experiments demonstrate that DRQ improves models quantized by several PTQ methods across dense and mixture‑of‑experts large language models, offering better downstream performance without extra inference cost.

By Yanlong Zhao, Xiaoyuan Cheng, Huihang Liu, Baihua He, Xinyu Zhang, Harrison Bo Hua Zhu, Wenlong Chen, Li Zeng, Zhuo Sun
arXiv Machine Learning
Oct 22

A Comprehensive Evaluation on Quantization Techniques for Large Language Models

The paper presents a comprehensive evaluation of post‑training quantization (PTQ) techniques for large language models (LLMs), addressing gaps in prior work by conducting fair, large‑scale experiments on text‑only LLMs. It decouples quantization into pre‑quantization transformation and error mitigation, systematically studies granularity, symmetry, and data formats (MXFP4, NVFP4), and finds that rotation‑based transformations and low‑rank compensation with GPTQ can improve performance, while finer granularity and asymmetric quantization benefit activations more than weights. The study highlights the importance of understanding component interactions and the impact of scaling factor formats on low‑bit quantization performance.

By Yutong Liu, Cairong Zhao, Guosheng Hu
Hugging Face Trending Papers
Oct 29

A Unified Hessian-Guided Framework for Post-Training Quantization

The paper introduces a unified Hessian-guided framework for post-training quantization of large language models. It derives a Hessian-based approximation of output changes due to joint weight and activation quantization, enabling simultaneous optimization of quantization ranges, linear transformations, and weight rounding under a single output-sensitivity objective. The method remains practical for modern models and consistently outperforms existing transformation and rounding techniques in both channel-wise and block-wise settings.

arXiv Machine Learning
Nov 2

Trainable Vector Quantization for Large Language Models

The paper introduces Trainable Vector Quantization (TVQ), a method that uses straight-through estimation to allow gradient descent optimization of vector-quantized large language models. TVQ reformulates VQ with trainable latent weights, enabling arbitrary objective minimization and is applied in a block-wise knowledge distillation framework to balance quality and training efficiency. Experiments show TVQ achieves state‑of‑the‑art low‑bit post‑training quantization, achieving near‑lossless 2‑bit quantization for LLaMA2‑70B with only a 1% accuracy drop on question‑answering benchmarks.

By Anastasiia Prutianova, Alina Bogdanova, Mikhail Shutov, Dmitrii Poimanov, Anatoly Gorkov, Ilia Zharikov