arXiv:2601. 07475v2 Announce Type: replace-cross Abstract: The emergence of fine-grained numerical formats like NVFP4 presents new opportunities for efficient Large Language Model (LLM) inference.
InfoQuant is a train‑free method that reshapes activation distributions to be more compatible with low‑bit quantizers. It uses Peak Suppression Orthogonal Transformation (PSOT) and adaptive outlier‑token selection to reduce numerical range while maintaining dispersion, thereby lowering quantization error. Across several large language model families, InfoQuant outperforms existing post‑training quantization and end‑to‑end training baselines, preserving 97% of floating‑point accuracy on average and narrowing the LLaMA‑2 13B performance gap by 42%.
arXiv:2606. 15652v1 Announce Type: new Abstract: 4-bit quantization significantly reduces the memory footprint and accelerates the inference of large language models (LLMs).
DualQuant introduces a dual‑scale, calibration‑free post‑training quantization framework that jointly optimizes weight‑reconstruction MSE and derives per‑channel activation rescalings, enabling effective 4‑bit weight and activation quantization without calibration data. The method employs a non‑differentiable optimization solved via variable splitting and block coordinate descent, supporting a wide range of formats including MX and NVFP4. Experiments on Llama and Qwen models show DualQuant matches or surpasses calibration‑based SmoothQuant and significantly outperforms other calibration‑free baselines, achieving up to 2.61× memory reduction and 2.46× throughput speedup on an RTX 4090.
arXiv:2607. 17733v1 Announce Type: cross Abstract: 4-bit quantization enables efficient LLM inference, but suffers from significant accuracy degradation due to outliers.
PRQuant introduces a training‑free, low‑overhead method for low‑bit quantization of linear layers by permuting input channels that cause the largest quantization error into contiguous tail blocks and precomputing residual weight sub‑tensors. The approach eliminates the need for online gathering during inference, converting scattered residual compensation into a regular tail‑augmented GEMM and thereby reducing latency. Experiments show that PRQuant lowers down‑projection reconstruction error and outperforms standard MXFP4 and other post‑training quantization baselines on five downstream benchmarks, improving accuracy by up to 1.24 points on Qwen3‑4B‑Instruct‑2507.
arXiv:2606. 07116v1 Announce Type: cross Abstract: Low-bit quantization has been widely adopted to accelerate the inference of large language models (LLMs) by significantly reducing computational cost and memory usage.
arXiv:2606. 26587v1 Announce Type: cross Abstract: Low-bit floating-point formats and semi-structured sparsity are increasingly supported by modern accelerators, yet combining them for LLM activation compression remains challenging: activations contain input-dependent outliers that dominate block scales in FP4 quantization, and directly applying N:M sparsity masks discards moderate values, coupling sparsification loss with quantization error.
arXiv:2606. 07618v1 Announce Type: cross Abstract: NVFP4 is a recently introduced hardware-supported FP4 format that improves the fidelity of 4-bit quantization through fine-grained block scales.
arXiv:2507. 23035v4 Announce Type: replace Abstract: Large language models (LLMs) have demonstrated impressive capabilities across a wide range of applications, but demand substantial memory and compute resources during inference.
arXiv:2603. 10444v2 Announce Type: replace-cross Abstract: FP4 training promises substantial memory and compute savings for large language models, but remains fragile because blockwise quantization is dictated by extreme activation magnitudes, which inflate dynamic range and compress long-tail signals.
The paper investigates low‑bit quantization for Multimodal Large Language Models (MLLMs), showing that MXFP8 retains near‑lossless performance while 4‑bit formats like MXFP4 and HiF4 cause significant degradation. It identifies activation quantization as the main source of this loss and introduces Residual Fallback Quantization (RFQ), a lightweight framework that adds a quantized residual pathway to improve activation fidelity without architectural changes. Experiments on Wan2.2 and Qwen3‑VL demonstrate that RFQ recovers much of the performance gap to BF16 baselines across generation and reasoning tasks.