arXiv Machine Learning

Beyond Scalar Sensitivity: Activation-Aware Mixed-Precision LLM Quantization with Cross-Layer Refinement

arXiv AI
Jun 4

dMX: Differentiable Mixed-Precision Assignment for Low-Precision Floating-Point Formats

arXiv:2606. 04115v1 Announce Type: cross Abstract: Quantizing large language models (LLMs) to low-precision floating-point representations is central to efficient deployment, yet applying a single bit-width uniformly across all layers is sub-optimal in terms of both performance and accuracy.

By Giuseppe Franco, Ian Colbert, Pablo Monteagudo-Lago, Felix Marty, Nicholas Fraser
arXiv AI
Oct 12

DualQuant: Dual-scale Calibration-free Weight-and-Activation Quantization for LLMs

DualQuant introduces a dual‑scale, calibration‑free post‑training quantization framework that jointly optimizes weight‑reconstruction MSE and derives per‑channel activation rescalings, enabling effective 4‑bit weight and activation quantization without calibration data. The method employs a non‑differentiable optimization solved via variable splitting and block coordinate descent, supporting a wide range of formats including MX and NVFP4. Experiments on Llama and Qwen models show DualQuant matches or surpasses calibration‑based SmoothQuant and significantly outperforms other calibration‑free baselines, achieving up to 2.61× memory reduction and 2.46× throughput speedup on an RTX 4090.

By Mahsa Salmani, Sayeh Sharify, Hesham Mostafa
arXiv AI
Aug 28

GAMMA: Global Bit Allocation for Mixed-Precision Models under Arbitrary Budgets

GAMMA is a post‑training framework that learns module‑wise precision preferences for mixed‑precision quantization of large language models. It optimizes a teacher‑forced hidden‑state reconstruction objective under an augmented Lagrangian constraint and then projects the learned preferences into exact budget‑feasible discrete assignments via integer programming. Because the learned preferences encode a stable sensitivity ranking, a single training run can be reused for any deployment budget, reducing per‑budget adaptation from hours to minutes and outperforming fixed‑precision baselines and search‑based methods on Llama and Qwen models.

By Zhangyang Yao, Haiyan Zhao, Haoyu Wang, Xu Han
Hugging Face Trending Papers
Oct 22

ProQuant: Preserving Expert Quality over Quantity in Ultra-Low-Bit Quantization of Mixture-of-Experts Models

ProQuant is a weight‑only post‑training quantization framework designed for Mixture‑of‑Experts large language models. It introduces PoP, which jointly prunes experts and assigns mixed‑precision bit‑widths to better use a tight bit budget, and RoCo, a routing‑consistent RMSNorm fine‑tune that mitigates routing drift caused by quantization errors. Experiments show ProQuant outperforms existing methods on a range of generative tasks.

arXiv Machine Learning
Jun 2

WINDQuant: Weight-Informed Neural Decision-Making for Global Mixed-Precision LLM Quantization

arXiv:2605. 26660v2 Announce Type: replace Abstract: Quantization is an effective approach to reduce the memory footprint and inference cost of large language models (LLMs), yet maintaining performance in the ultra-low-bit regime remains challenging.

By Phong Nam Huu Nguyen, Khoi M. Le, Cong-Duy T Nguyen, Anh Tuan Luu, Thong Thanh Nguyen, Tho Quan
arXiv Computation and Language
Nov 4

MoBiQuant: Mixture-of-Bits Quantization for Token-Adaptive Any-Precision LLM

MoBiQuant introduces a Mixture-of-Bits quantization framework that allows large language models to be inferred at varying precisions during runtime. It addresses the poor generalization of existing post‑training quantization methods by mitigating precision‑dependent outlier migration through a recursive residual quantization scheme and a token‑aware router that selects the optimal precision per token. Experiments demonstrate that MoBiQuant matches or exceeds single‑precision PTQ performance while providing significant memory savings and up to 1.34× throughput gains over current any‑precision approaches.

By Dongwei Wang, Jinhee Kim, Seokho Han, Denis Gudovskiy, Yohei Nakata, Tomoyuki Okuno, KhayTze Peong, Kang Eun Jeon, Jong Hwan Ko, Yiran Chen, Huanrui Yang
arXiv Machine Learning
Oct 23

FIGFLEX: A Mixed-Precision FP-INT GEMM Accelerator Co-Designed with Low-Bit LLM Quantization

FIGFLEX is a mixed‑precision accelerator co‑designed with a low‑bit LLM quantizer that performs floating‑point‑integer (FP‑INT) GEMM. It introduces a shift‑add, radix‑4 reformulation of FP‑INT multiplication that fuses multiply‑accumulate into a precision‑flexible partial‑product reduction, and a novel INT3 optimization that reclaims idle datapath leaves. The accompanying mixed‑precision quantizer assigns channel‑group bit‑widths based on GPTQ error, using power‑of‑two column scaling to keep activations out of the FP16 subnormal range, enabling the quantized model to run on FIGFLEX with the same accuracy as on a GPU while achieving higher throughput per area on 3‑bit mixed‑precision workloads.

By Faraz Tahmasebi, Haocheng Xu, Andrew Trauger, Yomal Perera, Zhenyu Tang, Nael Chowdhury, Sitao Huang
arXiv AI
Jun 4

Model-Preserving Adaptive Rounding

arXiv:2505. 22988v3 Announce Type: replace-cross Abstract: The goal of quantization is to produce a compressed model whose output distribution is as close to the original model's as possible.

By Albert Tseng, Zhaofeng Sun, Christopher De Sa