arXiv AI
OCGQuant: Outlier-Companion Grouping for NVFP4 Quantization
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Jul 7
ARCQuant: Boosting NVFP4 Quantization with Augmented Residual Channels for LLMs
arXiv:2601. 07475v2 Announce Type: replace-cross Abstract: The emergence of fine-grained numerical formats like NVFP4 presents new opportunities for efficient Large Language Model (LLM) inference.
arXiv Machine Learning
Oct 22
InfoQuant: Shaping Activation Distributions for Low-Bit LLM Quantization
InfoQuant is a train‑free method that reshapes activation distributions to be more compatible with low‑bit quantizers. It uses Peak Suppression Orthogonal Transformation (PSOT) and adaptive outlier‑token selection to reduce numerical range while maintaining dispersion, thereby lowering quantization error. Across several large language model families, InfoQuant outperforms existing post‑training quantization and end‑to‑end training baselines, preserving 97% of floating‑point accuracy on average and narrowing the LLaMA‑2 13B performance gap by 42%.
arXiv Machine Learning
Jun 16
MosaicQuant: Inlier-Outlier Disaggregation for Unified 4-Bit LLM Quantization
arXiv:2606. 15652v1 Announce Type: new Abstract: 4-bit quantization significantly reduces the memory footprint and accelerates the inference of large language models (LLMs).
arXiv AI
Oct 12
DualQuant: Dual-scale Calibration-free Weight-and-Activation Quantization for LLMs
DualQuant introduces a dual‑scale, calibration‑free post‑training quantization framework that jointly optimizes weight‑reconstruction MSE and derives per‑channel activation rescalings, enabling effective 4‑bit weight and activation quantization without calibration data. The method employs a non‑differentiable optimization solved via variable splitting and block coordinate descent, supporting a wide range of formats including MX and NVFP4. Experiments on Llama and Qwen models show DualQuant matches or surpasses calibration‑based SmoothQuant and significantly outperforms other calibration‑free baselines, achieving up to 2.61× memory reduction and 2.46× throughput speedup on an RTX 4090.
arXiv AI
Jul 21
MXSens: Sensitivity-Aware Mixed-Precision Quantization for Efficient LLM Inference
arXiv:2607. 17733v1 Announce Type: cross Abstract: 4-bit quantization enables efficient LLM inference, but suffers from significant accuracy degradation due to outliers.
arXiv Machine Learning
Sep 22
PRQuant: Permutation Residual Quantization for Low-Overhead Inference
PRQuant introduces a training‑free, low‑overhead method for low‑bit quantization of linear layers by permuting input channels that cause the largest quantization error into contiguous tail blocks and precomputing residual weight sub‑tensors. The approach eliminates the need for online gathering during inference, converting scattered residual compensation into a regular tail‑augmented GEMM and thereby reducing latency. Experiments show that PRQuant lowers down‑projection reconstruction error and outperforms standard MXFP4 and other post‑training quantization baselines on five downstream benchmarks, improving accuracy by up to 1.24 points on Qwen3‑4B‑Instruct‑2507.