Deriving Neural Scaling Laws from the statistics of natural language

Universal One-third Time Scaling in Learning Peaked Distributions

arXiv:2602. 03685v2 Announce Type: replace-cross Abstract: Training large language models (LLMs) is computationally expensive, partly because the loss exhibits slow power-law convergence whose origin remains debatable.

Scaling laws for neural language models

Neural Scaling Universality: If Exponents Are Fixed, Time to Understand Coefficients

arXiv:2606. 25008v1 Announce Type: new Abstract: Neural scaling laws describe how pre-training loss decays as power laws with training time, model size, and compute.

On the Nonlinearity of Learning Rate Scaling for LLM Training

arXiv:2606. 29158v1 Announce Type: cross Abstract: Learning-rate transfer can reduce the cost of training large language models: instead of sweeping learning rates at target scale, practitioners extrapolate from smaller runs.

Unifying Learning Dynamics and Generalization in Transformers Scaling Law

arXiv:2512. 22088v3 Announce Type: replace-cross Abstract: The scaling law, a cornerstone of Large Language Model (LLM) development, predicts improvements in model performance with increasing computational resources.

Context Staircase: Signature-Aligned Dynamics of Token Embeddings under Small Initialization

arXiv:2608.30315v1 Announce Type: new Abstract: Token embeddings are the basic representational units that connect discrete tokens with continuous computation in language models. Although modern lang...