arXiv Machine Learning
A Boundary-Layer Mechanism for One-Third Scaling in Online Softmax Classification
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Oct 23
A Fix for Softmax and Cross-Entropy
The paper identifies two numerical bugs in standard softmax, log‑softmax, and cross‑entropy implementations: Softmax Collapse, where a dominant logit causes its gradient to be rounded to zero, and Softmax Cancellation, where student and teacher logits that are equal still produce non‑zero gradients. The authors provide straightforward implementations that correct both issues. Their fix removes the previously observed "Slingshot" phenomenon, accelerates grokking with weight decay, prevents accuracy collapse without weight decay, keeps student networks aligned with teachers during distillation, and demonstrates that post‑memorization logit growth is bounded by an equilibrium between weight decay and Adam’s epsilon hyperparameter.
arXiv AI
Jun 2
Universal One-third Time Scaling in Learning Peaked Distributions
arXiv:2602. 03685v2 Announce Type: replace-cross Abstract: Training large language models (LLMs) is computationally expensive, partly because the loss exhibits slow power-law convergence whose origin remains debatable.
arXiv Statistics ML
6d ago
Depth Improves Training-Time Scaling for Difficult Tasks in Deep Linear Networks
arXiv:2610.35514v1 Announce Type: new Abstract: Neural scaling laws describe the predictable power-law dependence of model performance on training time, dataset size, and model scale, with the expone...
arXiv Machine Learning
Jun 25
Neural Scaling Universality: If Exponents Are Fixed, Time to Understand Coefficients
arXiv:2606. 25008v1 Announce Type: new Abstract: Neural scaling laws describe how pre-training loss decays as power laws with training time, model size, and compute.
arXiv Machine Learning
Sep 30
Optimizer-dependent training dynamics converge to the same one-third optimal data scaling
arXiv:2609.37745v1 Announce Type: new Abstract: Neural scaling, in which loss falls as a power law with training, is central to large language models, and one recent proposal is that a $1/3$ exponent...
arXiv AI
Jun 30
On the Nonlinearity of Learning Rate Scaling for LLM Training
arXiv:2606. 29158v1 Announce Type: cross Abstract: Learning-rate transfer can reduce the cost of training large language models: instead of sweeping learning rates at target scale, practitioners extrapolate from smaller runs.