arXiv Machine Learning By Dung Anh Hoang, Cuong Pham, Cuong Nguyen, Trung le, Jianfei Cai, Thanh-Toan Do

Beyond local reconstruction: Geometry-preserving output alignment for binary LLM quantization

Read the original on arXiv Machine Learning →

The paper introduces a geometry-preserving output alignment framework for 1‑bit post‑training quantization of large language models. It formulates output alignment as a constrained optimization that balances reconstruction quality with token similarity preservation, and proposes a geometry‑aware salient‑mask selection strategy to choose quantization settings that maintain token relations. Experiments on various LLM families and downstream tasks show consistent gains over existing 1‑bit PTQ methods.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv AI
Oct 31

A Unified Hessian-Guided Framework for Post-Training Quantization

The paper introduces a unified Hessian-guided framework for post‑training quantization of large language models. By deriving a Hessian‑based approximation of output changes, it jointly optimizes quantization ranges, linear transformations, and weight rounding under a single output‑sensitivity objective. The method remains practical for modern models and consistently outperforms existing transformation and rounding techniques in both channel‑wise and block‑wise settings.

By Mads Toftrup, Marco Federici, Boris van Breugel, Paul Whatmough, Markus Nagel
arXiv AI
Oct 9

When Lower Reconstruction Loss Hurts: Distributionally Robust Refinement for Low-Bit LLM Quantization

The paper examines weight-only post‑training quantization (PTQ) and finds that minimizing reconstruction loss does not always improve model performance, sometimes even harming it on calibration data. It shows that weights with low reconstruction loss on one distribution can perform worse when activation distributions shift. To address this, the authors introduce Distributionally Robust Quantization (DRQ), a refinement step that minimizes worst‑case reconstruction loss over a constrained set of activation distributions while keeping the quantization grid unchanged. Experiments demonstrate that DRQ improves models quantized by several PTQ methods across dense and mixture‑of‑experts large language models, offering better downstream performance without extra inference cost.

By Yanlong Zhao, Xiaoyuan Cheng, Huihang Liu, Baihua He, Xinyu Zhang, Harrison Bo Hua Zhu, Wenlong Chen, Li Zeng, Zhuo Sun
arXiv Machine Learning
Oct 22

A Comprehensive Evaluation on Quantization Techniques for Large Language Models

The paper presents a comprehensive evaluation of post‑training quantization (PTQ) techniques for large language models (LLMs), addressing gaps in prior work by conducting fair, large‑scale experiments on text‑only LLMs. It decouples quantization into pre‑quantization transformation and error mitigation, systematically studies granularity, symmetry, and data formats (MXFP4, NVFP4), and finds that rotation‑based transformations and low‑rank compensation with GPTQ can improve performance, while finer granularity and asymmetric quantization benefit activations more than weights. The study highlights the importance of understanding component interactions and the impact of scaling factor formats on low‑bit quantization performance.

By Yutong Liu, Cairong Zhao, Guosheng Hu
Hugging Face Trending Papers
Oct 29

A Unified Hessian-Guided Framework for Post-Training Quantization

The paper introduces a unified Hessian-guided framework for post-training quantization of large language models. It derives a Hessian-based approximation of output changes due to joint weight and activation quantization, enabling simultaneous optimization of quantization ranges, linear transformations, and weight rounding under a single output-sensitivity objective. The method remains practical for modern models and consistently outperforms existing transformation and rounding techniques in both channel-wise and block-wise settings.