arXiv Machine Learning

MSign: An Optimizer Preventing Training Instability in Large Language Models via Stable Rank Restoration

The paper introduces MSign, an optimizer designed to prevent training instability in large language models by restoring the stable rank of weight matrices. It identifies two precursors to gradient explosions—rapid stable rank decline and increased Jacobian alignment—and proves that these jointly cause exponential gradient growth. Experiments on models ranging from 5 M to 3 B parameters show that MSign stops training failures while adding less than 7.0% computational overhead.

arXiv Machine Learning
Jul 17

Stabilizing Native Low-Rank LLM Pretraining

arXiv:2602. 12429v2 Announce Type: replace Abstract: Foundation models have achieved remarkable success, yet their growing parameter counts pose significant computational and memory challenges.

By Paul Janson, Edouard Oyallon, Eugene Belilovsky
arXiv AI
Jun 2

Memory-Efficient LLM Training with Dynamic Sparsity: From Stability to Practical Scaling

arXiv:2606. 00888v1 Announce Type: cross Abstract: Dynamic Sparse Training (DST) offers a promising paradigm for improving the training and inference efficiency of deep neural networks; however, we find that in large language model training, DST can suffer from optimization instability, manifested as loss spikes after topology updates.

By Qiao Xiao, Boqian Wu, Patrik Okanovic, Tomasz Sternal, Maurice van Keulen, Elena Mocanu, Mykola Pechenizkiy, Decebal Constantin Mocanu, Torsten Hoefler
arXiv Machine Learning
Aug 31

Negligible in Size, Significant in Effect: On Scale Vectors in Large Language Models

The paper investigates the often-overlooked scale vectors in large language models, showing that despite their tiny size they are crucial for pre‑training performance. The authors provide theoretical insights that scale vectors mainly aid optimization rather than expressivity, and they analyze how weight decay affects different normalization layers. Building on these findings, they propose lightweight improvements—branch‑specific heterogeneity, better placement, and magnitude‑direction reparameterization—that consistently reduce loss across a range of model sizes and training settings.

By Mingze Wang, Shuchen Zhu, Yuxin Fang, Binghui Li, Kai Shen, Shu Zhong
arXiv AI
Oct 31

The Role of Quantization in Loss Spikes

The paper investigates loss spikes that occur late in language model pretraining, attributing them to delayed, terminal divergences in output logits—termed time‑bomb spikes. It shows that these spikes are preceded by rapidly growing quantization error in the LM head, and that reducing numerical precision speeds up the spikes while increasing precision slows them. The authors propose replacing AdamW with SGD with momentum on the LM head to suppress logit growth, and find that various mitigation techniques reduce quantization error, indicating a common numerical cause for stability.

By Zaiqian Chen, Aditya Somasundaram, Sophie Li, Sean McLeish, John Kirchenbauer, Tom Goldstein, Micah Goldblum
arXiv Machine Learning
Oct 12

Weight Clustering Reveals the Importance of Relative Rank in Large Language Models

The study demonstrates that large language models can be compressed by clustering weight matrices into only 16–64 shared values without retraining, preserving strong accuracy. Fine‑tuning the cluster centroids recovers most of the remaining accuracy loss. Analysis reveals that maintaining the relative rank of weights and their overall variance is crucial for performance, while early layers and spatial geometry of cluster assignments also play significant roles.

By Borja Aizpurua, Sukhbinder Singh, Rom\'an Or\'us
Hugging Face Trending Papers
Nov 9

KiteNorm: Variance Regularisation for Stable and Scalable Post-LN Transformers

KiteNorm is a post‑LayerNorm training method that stabilises decoder‑only Transformers by constraining hidden‑state variance through residual scaling and variance regularisation. It addresses the instability of existing post‑norm approaches, especially at large learning rates, and outperforms strong Pre‑LayerNorm baselines on models up to 1.3 B parameters. Ablation studies confirm that each component of KiteNorm is essential for its full stability and performance gains.

arXiv Machine Learning
Aug 27

Spectral Allocation: Why Muon Outperforms Adam, and How to Improve Muon

The paper investigates why the orthogonal optimiser Muon outperforms Adam in large language model pretraining by analysing the spectral properties of Transformer loss landscapes. It finds that Muon’s momentum buffers exhibit an anisotropic spectral profile with a volatile head and a tolerant bulk, enabling larger effective step sizes. Building on this insight, the authors propose Spectral‑Aware Muon (SAMuon) and a lightweight variant, which adjust the bulk scaling while keeping the head unchanged, achieving 13–24 % fewer training tokens than Muon without extra FLOPs.

By Xiaodong Wu, Wenyi Yu, Chao Zhang, Philip Woodland
arXiv AI
Sep 3

Scalable Kronecker-Fisher Approximation: Efficient Hessian Analysis for Billion-Parameter Language Models Compression

The paper introduces a scalable Kronecker-based approximation that captures cross-layer interactions without storing the full Fisher matrix, making Hessian analysis feasible for billion-parameter language models. It identifies consistent vulnerability patterns, notably that value projection layers are the most sensitive and exhibit strong cross-layer correlations across various model families. Experiments on quantization, sparsification, inter-layer corruption, and fine-tuning show that the approximation correlates strongly with performance degradation and recovery, providing a practical tool for identifying fragile components and guiding compression and optimization strategies.

By Viacheslav Yusupov, Daria Cherniuk, Evgeny Frolov