arXiv:2607. 23047v1 Announce Type: cross Abstract: Mixed-precision quantization improves the accuracy of post-training quantization by allocating higher bitwidths to sensitive layers, but existing methods solve the allocation for a single fixed memory budget.
By Ashitabh Misra, Madhav Agrawal, Arham Jain, Tarek Abdelzaher
arXiv:2606. 04115v1 Announce Type: cross Abstract: Quantizing large language models (LLMs) to low-precision floating-point representations is central to efficient deployment, yet applying a single bit-width uniformly across all layers is sub-optimal in terms of both performance and accuracy.
By Giuseppe Franco, Ian Colbert, Pablo Monteagudo-Lago, Felix Marty, Nicholas Fraser
DualQuant introduces a dual‑scale, calibration‑free post‑training quantization framework that jointly optimizes weight‑reconstruction MSE and derives per‑channel activation rescalings, enabling effective 4‑bit weight and activation quantization without calibration data. The method employs a non‑differentiable optimization solved via variable splitting and block coordinate descent, supporting a wide range of formats including MX and NVFP4. Experiments on Llama and Qwen models show DualQuant matches or surpasses calibration‑based SmoothQuant and significantly outperforms other calibration‑free baselines, achieving up to 2.61× memory reduction and 2.46× throughput speedup on an RTX 4090.
By Mahsa Salmani, Sayeh Sharify, Hesham Mostafa
arXiv:2608.30564v1 Announce Type: cross
Abstract: Mixed-precision quantization (MPQ) assigns a different bitwidth to each linear layer of a large language model (LLM) to minimize the quantization-ind...
By Deokjae Lee, Sihun Chu, Hyun Oh Song
GAMMA is a post‑training framework that learns module‑wise precision preferences for mixed‑precision quantization of large language models. It optimizes a teacher‑forced hidden‑state reconstruction objective under an augmented Lagrangian constraint and then projects the learned preferences into exact budget‑feasible discrete assignments via integer programming. Because the learned preferences encode a stable sensitivity ranking, a single training run can be reused for any deployment budget, reducing per‑budget adaptation from hours to minutes and outperforming fixed‑precision baselines and search‑based methods on Llama and Qwen models.
By Zhangyang Yao, Haiyan Zhao, Haoyu Wang, Xu Han
arXiv:2610.14745v1 Announce Type: new
Abstract: Extreme low-bit weight-only quantization is an effective approach to reducing the deployment costs of large language models. Below 3 bits, scalar quant...
By Xianglong Yan, Zenan Wu, Kewei Zhang, Kaicheng Yang, Yulun Zhang
ProQuant is a weight‑only post‑training quantization framework designed for Mixture‑of‑Experts large language models. It introduces PoP, which jointly prunes experts and assigns mixed‑precision bit‑widths to better use a tight bit budget, and RoCo, a routing‑consistent RMSNorm fine‑tune that mitigates routing drift caused by quantization errors. Experiments show ProQuant outperforms existing methods on a range of generative tasks.
arXiv:2605. 26660v2 Announce Type: replace Abstract: Quantization is an effective approach to reduce the memory footprint and inference cost of large language models (LLMs), yet maintaining performance in the ultra-low-bit regime remains challenging.
By Phong Nam Huu Nguyen, Khoi M. Le, Cong-Duy T Nguyen, Anh Tuan Luu, Thong Thanh Nguyen, Tho Quan
MoBiQuant introduces a Mixture-of-Bits quantization framework that allows large language models to be inferred at varying precisions during runtime. It addresses the poor generalization of existing post‑training quantization methods by mitigating precision‑dependent outlier migration through a recursive residual quantization scheme and a token‑aware router that selects the optimal precision per token. Experiments demonstrate that MoBiQuant matches or exceeds single‑precision PTQ performance while providing significant memory savings and up to 1.34× throughput gains over current any‑precision approaches.
By Dongwei Wang, Jinhee Kim, Seokho Han, Denis Gudovskiy, Yohei Nakata, Tomoyuki Okuno, KhayTze Peong, Kang Eun Jeon, Jong Hwan Ko, Yiran Chen, Huanrui Yang
FIGFLEX is a mixed‑precision accelerator co‑designed with a low‑bit LLM quantizer that performs floating‑point‑integer (FP‑INT) GEMM. It introduces a shift‑add, radix‑4 reformulation of FP‑INT multiplication that fuses multiply‑accumulate into a precision‑flexible partial‑product reduction, and a novel INT3 optimization that reclaims idle datapath leaves. The accompanying mixed‑precision quantizer assigns channel‑group bit‑widths based on GPTQ error, using power‑of‑two column scaling to keep activations out of the FP16 subnormal range, enabling the quantized model to run on FIGFLEX with the same accuracy as on a GPU while achieving higher throughput per area on 3‑bit mixed‑precision workloads.
By Faraz Tahmasebi, Haocheng Xu, Andrew Trauger, Yomal Perera, Zhenyu Tang, Nael Chowdhury, Sitao Huang
Weight-only quantization keeps LLM activations in floating point and stores weights as low-bit integers, turning the dominant workload into floating-point-integer (FP-INT) general matrix multiplicatio...
arXiv:2505. 22988v3 Announce Type: replace-cross Abstract: The goal of quantization is to produce a compressed model whose output distribution is as close to the original model's as possible.
By Albert Tseng, Zhaofeng Sun, Christopher De Sa