arXiv Machine Learning September 3, 2026 By Robert Hu, Carlo Luschi, Paul Balanca

UE5M3 FP4 Block Scaling for Stable Language Model Pretraining

Read the original on arXiv Machine Learning →

The paper introduces a new 4‑bit floating‑point (FP4) pretraining approach that pairs E2M1 payloads with unsigned E5M3 block scales, enabling periodic tensor scaling and selective stochastic rounding while eliminating the randomized Hadamard transform. Using this method, the authors pretrained a Nemotron‑H 8B model on nearly 190 billion tokens, achieving lower training and validation losses compared to NVIDIA’s Transformer Engine. The approach also improves inference performance and demonstrates a 21.2 % increase in token throughput when certain optimizations are removed.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Hugging Face Trending Papers

Sep 2

UE5M3 FP4 Block Scaling for Stable Language Model Pretraining

Stable 4-bit floating-point (FP4) pretraining is difficult because the E2M1 payload represents only a narrow range of magnitudes. NVIDIA's Transformer Engine \nv{} recipe addresses this with current-tensor scaling, a randomized Hadamard transform (RHT), and bfloat16 (BF16) final layers, adding work outside the FP4 matrix multiplications.

llms efficiency benchmarks

arXiv AI

Jun 19

Rethinking Shrinkage Bias in LLM FP4 Pretraining: Geometric Origin, Systemic Impact, and UFP4 Recipe

arXiv:2606. 20381v1 Announce Type: new Abstract: FP4 training promises substantial reductions in memory and computation cost for LLM pretraining, yet current FP4 hardware paths and recipes, including NVIDIA Blackwell/Rubin-class systems and AMD MI350-series GPUs, remain centered on E2M1 data elements.

By Qian Zhao, Kunlong Chen, Changxin Tian, Zhonghui Jiang, Haitao Zhang, Chaofan Yu, Peijie Jiang, Mingliang Gong, Jia Liu, Ziqi Liu, Zhiqiang Zhang, Jun Zhou

llms efficiency benchmarks safety

arXiv AI

Aug 13

Pretraining large language models with MXFP4 on Native FP4 Hardware

arXiv:2605. 09825v4 Announce Type: replace-cross Abstract: Why does full-pipeline FP4 training of large language models often diverge, even when forward activations and activation gradients remain stable?

By Musa Cim, Sarthak Arora, Poovaiah Palangappa, Miro Hodak, Ravi Dwivedula, Meena Arunachalam, Mahmut Taylan Kandemir

llms efficiency

arXiv AI

Jun 9

ScaleSweep: Accurate NVFP4 Post-Training Quantization of LLMs via Block Scale Initialization

arXiv:2606. 07618v1 Announce Type: cross Abstract: NVFP4 is a recently introduced hardware-supported FP4 format that improves the fidelity of 4-bit quantization through fine-grained block scales.

By Li Lin, Xiaojun Wan

llms efficiency

arXiv Computation and Language

Aug 31

H-Scale: Hessian-Guided Scale Refinement for NVFP4 Sub-Byte LLM Inference

The paper introduces H-Scale, a lightweight post-processing technique for refining per-group scaling factors in NVFP4 quantized large language models. By using a diagonal second-order proxy from calibration activations, H-Scale selects hardware-valid scales that directly target layer output perturbation rather than just weight reconstruction error. Experiments on mainstream LLMs show that H-Scale improves NVFP4 baselines and brings several variants closer to BF16 performance without adding inference overhead.

By Hao Yu, Zheng Li, Dayiheng Liu, Jianwei Zhang

llms efficiency

arXiv AI

Jul 29

Stable FP4 Training via Transposition-Invariant Block Quantization

arXiv:2607. 24953v1 Announce Type: cross Abstract: Reducing training precision is a key lever for improving the e ciency of large language model (LLM) training, but pushing beyond FP8 to 4-bit oating point (FP4) remains challenging due to instability during optimization.

By Mehdi Rahimifar, Amin Darabi, Mehran Taghian Jazi, Xing Huang, Yao Wang, Zhijun Tu, Yufei Cui, Yunke Peng, Hongliang Li

llms efficiency

One email a morning, machine-written

Related stories

UE5M3 FP4 Block Scaling for Stable Language Model Pretraining

Rethinking Shrinkage Bias in LLM FP4 Pretraining: Geometric Origin, Systemic Impact, and UFP4 Recipe

Pretraining large language models with MXFP4 on Native FP4 Hardware

ScaleSweep: Accurate NVFP4 Post-Training Quantization of LLMs via Block Scale Initialization

H-Scale: Hessian-Guided Scale Refinement for NVFP4 Sub-Byte LLM Inference

Stable FP4 Training via Transposition-Invariant Block Quantization