arXiv Machine Learning July 14, 2026 By Zekai Shang

The Silent Freeze: Predicting When Low-Precision Training Stops Learning

Read the original on arXiv Machine Learning →

arXiv:2607. 09800v1 Announce Type: new Abstract: Training in reduced floating-point precision can silently halt learning: when a gradient-descent weight update falls below half the unit in the last place (ULP) of the weight, it rounds away and that coordinate freezes while its gradient is still nonzero.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv Machine Learning.

llms

arXiv AI

Jun 16

Why Low-Precision Transformer Training Fails: An Analysis on Flash Attention

arXiv:2510. 04212v4 Announce Type: replace-cross Abstract: The pursuit of computational efficiency has driven the adoption of low-precision formats for training transformer models.

By Haiquan Qiu, Quanming Yao

llms efficiency safety

arXiv Machine Learning

Aug 7

Threshold-Based Early Stopping of Accumulations in Neural Networks with Binary Activation

arXiv:2608. 06177v1 Announce Type: new Abstract: Binary neural networks are very attractive for constrained deployment, enabling small footprint and low-power inference.

By Quentin Luquet de Saint-Germain, Massil Ait Abdeslam, Jean Pierre David

arXiv AI

Jul 28

Scale Weight Decay and Train Better

arXiv:2607. 23777v1 Announce Type: cross Abstract: The discovery of scaling laws has motivated training neural networks on ever increasing quantities of data.

By Anuj Apte

safety

arXiv AI

Jul 22

Federated Lightweight Fine-Tuning

arXiv:2607. 18343v1 Announce Type: cross Abstract: Federated fine-tuning is bottlenecked by communication: FedAvg and pseudo-gradient schemes transmit a payload that scales with the model, and gradient compression shrinks it by only a constant factor.

By Radhakrishna Achanta, Will Reed

fine-tuning

arXiv Machine Learning

Jul 30

Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunities

arXiv:2505. 01043v2 Announce Type: replace Abstract: Large language models (LLMs) have achieved impressive performance across various domains.

By Zhiwei Hao, Jianyuan Guo, Li Shen, Yong Luo, Han Hu, Guoxia Wang, Dianhai Yu, Yonggang Wen, Dacheng Tao

llms efficiency

arXiv Machine Learning

Jul 30

HiFloat4 Format for End-To-End Reinforcement Learning Post-Training of Large Language Models

arXiv:2607. 26515v1 Announce Type: new Abstract: We present, to our knowledge, the first end-to-end FP4 RL post-training, in which both the rollout and training policies, including their forward and backward passes, operate at 4-bit precision.

By Hei Yi Mak, Shadan Golestan, Hoang Le, Mehran Taghian Jazi, Yunke Peng, Yaoyuan Wang, Yao Wang, Junsong Wang, Tianchi Hu, Fengchen He, Guipeng Hu, Tanzila Rahman, Anandharaju Durai Raju

llms reinforcement-learning efficiency

Related stories

Why Low-Precision Transformer Training Fails: An Analysis on Flash Attention

Threshold-Based Early Stopping of Accumulations in Neural Networks with Binary Activation

Scale Weight Decay and Train Better

Federated Lightweight Fine-Tuning

Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunities

HiFloat4 Format for End-To-End Reinforcement Learning Post-Training of Large Language Models