DualQuant introduces a dual‑scale, calibration‑free post‑training quantization framework that jointly optimizes weight‑reconstruction MSE and derives per‑channel activation rescalings, enabling effective 4‑bit weight and activation quantization without calibration data. The method employs a non‑differentiable optimization solved via variable splitting and block coordinate descent, supporting a wide range of formats including MX and NVFP4. Experiments on Llama and Qwen models show DualQuant matches or surpasses calibration‑based SmoothQuant and significantly outperforms other calibration‑free baselines, achieving up to 2.61× memory reduction and 2.46× throughput speedup on an RTX 4090.
arXiv:2606. 07618v1 Announce Type: cross Abstract: NVFP4 is a recently introduced hardware-supported FP4 format that improves the fidelity of 4-bit quantization through fine-grained block scales.
arXiv:2601. 07475v2 Announce Type: replace-cross Abstract: The emergence of fine-grained numerical formats like NVFP4 presents new opportunities for efficient Large Language Model (LLM) inference.
arXiv:2606. 26587v1 Announce Type: cross Abstract: Low-bit floating-point formats and semi-structured sparsity are increasingly supported by modern accelerators, yet combining them for LLM activation compression remains challenging: activations contain input-dependent outliers that dominate block scales in FP4 quantization, and directly applying N:M sparsity masks discards moderate values, coupling sparsification loss with quantization error.
The paper investigates low‑bit quantization for Multimodal Large Language Models (MLLMs), showing that MXFP8 retains near‑lossless performance while 4‑bit formats like MXFP4 and HiF4 cause significant degradation. It identifies activation quantization as the main source of this loss and introduces Residual Fallback Quantization (RFQ), a lightweight framework that adds a quantized residual pathway to improve activation fidelity without architectural changes. Experiments on Wan2.2 and Qwen3‑VL demonstrate that RFQ recovers much of the performance gap to BF16 baselines across generation and reasoning tasks.
arXiv:2606. 04238v1 Announce Type: cross Abstract: Aggressive weight quantization to 2-bit precision offers substantial throughput and memory gains for large language model (LLM) inference, but typically incurs severe accuracy degradation.
arXiv:2606. 13054v1 Announce Type: cross Abstract: Large language models (LLMs) exhibit exceptional general language processing capabilities, but their memory and compute costs hinder deployment.
The paper introduces H-Scale, a lightweight post-processing technique for refining per-group scaling factors in NVFP4 quantized large language models. By using a diagonal second-order proxy from calibration activations, H-Scale selects hardware-valid scales that directly target layer output perturbation rather than just weight reconstruction error. Experiments on mainstream LLMs show that H-Scale improves NVFP4 baselines and brings several variants closer to BF16 performance without adding inference overhead.
The paper introduces CanonQ, a unified quantization‑aware training framework that separates source canonicalization from task‑aware adaptation to enable extreme low‑bit LLM compression. By applying fixed rotations and energy normalization, CanonQ maps heterogeneous tensors to canonical coordinates, allowing frozen Gaussian‑reference codebooks to be reused across layers and models. Joint training then adapts the network to the coupled errors of weight, activation, and KV cache quantization, achieving significant performance gains—up to 14.28× lower perplexity and 57.9% higher zero‑shot accuracy on LLaMA models, and substantial improvements on instruction‑tuned MobileLLM‑Pro‑1B. "whyItMatters":"The framework demonstrates that carefully designed canonicalization and joint training can dramatically improve the efficiency and accuracy of low‑bit quantized large language models, offering a practical path to deploy powerful LLMs with reduced computational and memory footprints."
arXiv:2608. 11045v1 Announce Type: new Abstract: ReRound (Reconstructive Rounding) is a post-training quantization method that addresses the midpoint ambiguity inherent in standard round-to-nearest (RTN) schemes when quantizing weights near the centers of quantization intervals.
arXiv:2605. 08692v2 Announce Type: replace Abstract: Post-training weight-only quantization to 4 bits is widely used to reduce the memory and compute costs of large language model inference.
arXiv:2605.11222v2 Announce Type: replace Abstract: Quantization is an effective strategy to reduce the storage and computation footprint of large language models (LLMs). Post-training quantization (...
arXiv:2607. 17733v1 Announce Type: cross Abstract: 4-bit quantization enables efficient LLM inference, but suffers from significant accuracy degradation due to outliers.