arXiv Machine Learning June 2, 2026 By Andrei Panferov, Erik Schultheis, Soroush Tabesh, Dan Alistarh

Quartet II: Accurate LLM Pre-Training in NVFP4 by Improved Unbiased Gradient Estimation

Read the original on arXiv Machine Learning →

arXiv:2601. 22813v2 Announce Type: replace Abstract: The NVFP4 lower-precision format, supported in hardware by NVIDIA Blackwell GPUs, promises to allow, for the first time, end-to-end fully-quantized pre-training of massive models such as LLMs.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv Machine Learning.

arXiv AI

Jun 9

ScaleSweep: Accurate NVFP4 Post-Training Quantization of LLMs via Block Scale Initialization

arXiv:2606. 07618v1 Announce Type: cross Abstract: NVFP4 is a recently introduced hardware-supported FP4 format that improves the fidelity of 4-bit quantization through fine-grained block scales.

By Li Lin, Xiaojun Wan

llms efficiency

arXiv AI

Jul 7

ARCQuant: Boosting NVFP4 Quantization with Augmented Residual Channels for LLMs

arXiv:2601. 07475v2 Announce Type: replace-cross Abstract: The emergence of fine-grained numerical formats like NVFP4 presents new opportunities for efficient Large Language Model (LLM) inference.

By Haoqian Meng, Yilun Luo, Yafei Zhao, Wenyuan Liu, Peng Zhang, Xindian Ma

llms efficiency benchmarks

arXiv AI

Jun 19

Rethinking Shrinkage Bias in LLM FP4 Pretraining: Geometric Origin, Systemic Impact, and UFP4 Recipe

arXiv:2606. 20381v1 Announce Type: new Abstract: FP4 training promises substantial reductions in memory and computation cost for LLM pretraining, yet current FP4 hardware paths and recipes, including NVIDIA Blackwell/Rubin-class systems and AMD MI350-series GPUs, remain centered on E2M1 data elements.

By Qian Zhao, Kunlong Chen, Changxin Tian, Zhonghui Jiang, Haitao Zhang, Chaofan Yu, Peijie Jiang, Mingliang Gong, Jia Liu, Ziqi Liu, Zhiqiang Zhang, Jun Zhou

llms efficiency benchmarks safety

arXiv AI

Jun 15

The Curse and Blessing of Mean Bias in FP4-Quantized LLM Training

arXiv:2603. 10444v2 Announce Type: replace-cross Abstract: FP4 training promises substantial memory and compute savings for large language models, but remains fragile because blockwise quantization is dictated by extreme activation magnitudes, which inflate dynamic range and compress long-tail signals.

By Hengjie Cao, Zhendong Huang, Mengyi Chen, Yifeng Yang, Fang Dong, Anrui Chen, Ruijun Huang, Xin Zhang, Mingzhi Dong, Yujiang Wang, Jinlong Hou, Qin Lv, Robert P. Dick, Yuan Cheng, Tun Lu, Fan Yang, Yixuan Chen, Li Shang

llms efficiency safety

arXiv AI

Jul 29

Stable FP4 Training via Transposition-Invariant Block Quantization

arXiv:2607. 24953v1 Announce Type: cross Abstract: Reducing training precision is a key lever for improving the e ciency of large language model (LLM) training, but pushing beyond FP8 to 4-bit oating point (FP4) remains challenging due to instability during optimization.

By Mehdi Rahimifar, Amin Darabi, Mehran Taghian Jazi, Xing Huang, Yao Wang, Zhijun Tu, Yufei Cui, Yunke Peng, Hongliang Li

llms efficiency

arXiv Machine Learning

Jul 7

DynamiQ: Accelerating Gradient Synchronization using Compressed Multi-hop All-reduce

arXiv:2602. 08923v2 Announce Type: replace Abstract: Multi-hop all-reduce is the de facto backbone of large model training.

By Wenchen Han, Shay Vargaftik, Michael Mitzenmacher, Ran Ben Basat

llms efficiency benchmarks

Related stories

ScaleSweep: Accurate NVFP4 Post-Training Quantization of LLMs via Block Scale Initialization

ARCQuant: Boosting NVFP4 Quantization with Augmented Residual Channels for LLMs

Rethinking Shrinkage Bias in LLM FP4 Pretraining: Geometric Origin, Systemic Impact, and UFP4 Recipe

The Curse and Blessing of Mean Bias in FP4-Quantized LLM Training

Stable FP4 Training via Transposition-Invariant Block Quantization

DynamiQ: Accelerating Gradient Synchronization using Compressed Multi-hop All-reduce