The paper examines weight-only post‑training quantization (PTQ) and finds that minimizing reconstruction loss does not always improve model performance, sometimes even harming it on calibration data. It shows that weights with low reconstruction loss on one distribution can perform worse when activation distributions shift. To address this, the authors introduce Distributionally Robust Quantization (DRQ), a refinement step that minimizes worst‑case reconstruction loss over a constrained set of activation distributions while keeping the quantization grid unchanged. Experiments demonstrate that DRQ improves models quantized by several PTQ methods across dense and mixture‑of‑experts large language models, offering better downstream performance without extra inference cost.
By Yanlong Zhao, Xiaoyuan Cheng, Huihang Liu, Baihua He, Xinyu Zhang, Harrison Bo Hua Zhu, Wenlong Chen, Li Zeng, Zhuo Sun
arXiv:2606. 04349v1 Announce Type: cross Abstract: Conventional Post-Training Quantization (PTQ) methods struggle with 4-bit Omni-modal Large Language Models (OLLMs) due to the extreme distribution heterogeneity and disparate outlier patterns across modalities.
By Yue Wu, Changyuan Wang, Zixuan Wang, Shilin Ma, Yansong Tang
Large language models (LLMs) exhibit exceptional general language processing capabilities, but their memory and compute costs hinder deployment. Ternarization has emerged as a promising compression technique, offering significant reductions in model size and inference complexity.
arXiv:2507.17417v4 Announce Type: replace
Abstract: For large language models (LLMs), post-training quantization (PTQ) significantly reduces memory and computational overhead. Despite rapid advanceme...
By Yutong Liu, Cairong Zhao, Guosheng Hu
arXiv:2606. 13054v1 Announce Type: cross Abstract: Large language models (LLMs) exhibit exceptional general language processing capabilities, but their memory and compute costs hinder deployment.
By Zhixiong Zhao, Zukang Xu, Zhixuan Chen, Xing Hu, Zhe Jiang, Dawei Yang
DualQuant introduces a dual‑scale, calibration‑free post‑training quantization framework that jointly optimizes weight‑reconstruction MSE and derives per‑channel activation rescalings, enabling effective 4‑bit weight and activation quantization without calibration data. The method employs a non‑differentiable optimization solved via variable splitting and block coordinate descent, supporting a wide range of formats including MX and NVFP4. Experiments on Llama and Qwen models show DualQuant matches or surpasses calibration‑based SmoothQuant and significantly outperforms other calibration‑free baselines, achieving up to 2.61× memory reduction and 2.46× throughput speedup on an RTX 4090.
By Mahsa Salmani, Sayeh Sharify, Hesham Mostafa