The paper introduces Hardware‑Aware FP4 FlashAttention‑4, a method that leverages NVIDIA’s Blackwell 4‑bit floating‑point (FP4) tensor cores for attention mechanisms. It presents two key techniques: Direct‑P, which maps attention scores directly to FP4 probabilities for noncausal inference, achieving up to 2.13× the bfloat16 forward throughput on an NVIDIA GB200; and a causal path that reconstructs probabilities from quantized queries and keys while using 8‑bit floating‑point (FP8) gradients, accelerating a full single‑GPU 8‑billion‑parameter update by up to 1.14×. The authors also note that distributed training with matched FP8 probabilities and values diverges for every tested MXFP4 probability/value trajectory.
By Robert Hu
arXiv:2603. 00040v3 Announce Type: replace-cross Abstract: Achieving reliable 4-bit attention is a prerequisite for end-to-end FP4 computation on emerging FP4-capable GPUs, yet attention remains the main obstacle due to FP4's tiny dynamic range and attention's heavy-tailed activations.
By Peiyuan Zhang, Matthew Noto, Wenxuan Tan, Chengquan Jiang, Will Lin, Wei Zhou, Hao Zhang
arXiv:2601. 22813v2 Announce Type: replace Abstract: The NVFP4 lower-precision format, supported in hardware by NVIDIA Blackwell GPUs, promises to allow, for the first time, end-to-end fully-quantized pre-training of massive models such as LLMs.
By Andrei Panferov, Erik Schultheis, Soroush Tabesh, Dan Alistarh
arXiv:2607. 24953v1 Announce Type: cross Abstract: Reducing training precision is a key lever for improving the e ciency of large language model (LLM) training, but pushing beyond FP8 to 4-bit oating point (FP4) remains challenging due to instability during optimization.
By Mehdi Rahimifar, Amin Darabi, Mehran Taghian Jazi, Xing Huang, Yao Wang, Zhijun Tu, Yufei Cui, Yunke Peng, Hongliang Li
Stable 4-bit floating-point (FP4) pretraining is difficult because the E2M1 payload represents only a narrow range of magnitudes. NVIDIA's Transformer Engine \nv{} recipe addresses this with current-tensor scaling, a randomized Hadamard transform (RHT), and bfloat16 (BF16) final layers, adding work outside the FP4 matrix multiplications.
arXiv:2606. 15682v1 Announce Type: new Abstract: Large Reasoning Models (LRMs) achieve strong problem-solving through long chain-of-thought, but their deployment is constrained by the high cost of full-precision inference and growing KV cache footprints.
By Janghwan Lee, Sihwa Lee, Jinseok Kim, Yongjik Kim, Jieun Lim, Jinwook Oh, Jungwook Choi