← Back to all news
arXiv Machine Learning By Ran Tian, Courtney Paquette

Simulating Training Spikes of Deep Neural Networks by Layer-wise Random Feature Models

Read the original on arXiv Machine Learning →

The paper investigates intermittent spikes in loss observed during the pre‑training of large language models. It demonstrates that simple quadratic models with heavy‑tailed, power‑law distributed inputs can reproduce these spikes, and introduces a theoretical framework and simulation algorithm to model per‑layer training dynamics of deep neural networks. In a case study using an Adam configuration with β2 close to 1, the simulation predicts spiky behavior from a checkpoint and suggests a β2 schedule that empirically improves performance.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
Jun 2

Memory-Efficient LLM Training with Dynamic Sparsity: From Stability to Practical Scaling

arXiv:2606. 00888v1 Announce Type: cross Abstract: Dynamic Sparse Training (DST) offers a promising paradigm for improving the training and inference efficiency of deep neural networks; however, we find that in large language model training, DST can suffer from optimization instability, manifested as loss spikes after topology updates.

By Qiao Xiao, Boqian Wu, Patrik Okanovic, Tomasz Sternal, Maurice van Keulen, Elena Mocanu, Mykola Pechenizkiy, Decebal Constantin Mocanu, Torsten Hoefler
llmsefficiency
More like this →
arXiv Machine Learning
Oct 9

Grokking or Glitching? How Low-Precision Drives Slingshot Loss Spikes

Deep neural networks can exhibit periodic loss spikes known as the "Slingshot Mechanism" during long‑term training. This paper shows that such spikes arise from finite‑precision floating‑point arithmetic: when the correct‑class logit becomes much larger than the others, the gradient for that class is rounded to zero while others remain nonzero, breaking the zero‑sum constraint and creating a positive feedback loop that inflates feature and classifier norms. The authors term this Numerical Feature Inflation (NFI) and demonstrate that it explains the rapid norm growth, reappearance of gradients, and loss spikes, while also showing that partial absorption can still cause hidden norm growth without visible spikes.

By Hanqing Liu, Jianjun Cao, Yuanze Li, Zijian Zhou
More like this →
arXiv Machine Learning
Jul 2

Prototype Language Models

arXiv:2607. 00510v1 Announce Type: new Abstract: Knowing which training examples drive outputs is fundamental to auditing, correcting, and understanding language models, yet for modern LLMs this remains expensive, approximate, and largely post-hoc.

By Dan Ley, Giang Nguyen, Himabindu Lakkaraju, Julius Adebayo
llmsfine-tuning
More like this →