arXiv Machine Learning
Data Predictability Shapes Weibull Weight-Scale Growth in Transformer Training
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Sep 24
Does Step Law Transfer to Small-Scale Language Models? An Empirical Recalibration Below 59M Parameters
arXiv:2609.27581v1 Announce Type: new Abstract: Step Law gives power-law formulas for the optimal peak learning rate eta* and batch size B* when pre-training language models. It was calibrated on mod...
arXiv Machine Learning
Sep 23
Practical Scaling Laws: Converting Compute into Performance in a Data-Constrained World
The paper introduces a new closed‑form scaling law that extends Chinchilla’s original formula to handle data‑constrained regimes. It decomposes loss into undercapacity, undertraining, and overfitting components, saturating between an irreducible loss and an uninformed baseline. The authors validate the model on diverse architectures and domains, achieving state‑of‑the‑art RMSE across multiple LLM scaling‑law grids and enabling cost‑aware training allocations.
arXiv Machine Learning
Jun 19
Weibull Weight-Scale Parameter Evolution under AdamW Training Dynamics
arXiv:2606. 19367v1 Announce Type: new Abstract: Building on a two-parameter Weibull framework for diagnosing transformer weight distributions, we study why the Weibull weight-scale parameter $\lambda$ grows, overshoots, and then relaxes during AdamW training.
arXiv Machine Learning
Jul 27
Scaling Laws for Classical Machine Learning on Tabular Data: A Benchmark Study
arXiv:2607. 21866v1 Announce Type: new Abstract: Prior classical-ML learning-curve work fits power laws to tree, linear, and kernel models on tabular data, but at small scale: typically one curve, one team, a handful of cells.
arXiv AI
Jun 11
Unifying Learning Dynamics and Generalization in Transformers Scaling Law
arXiv:2512. 22088v3 Announce Type: replace-cross Abstract: The scaling law, a cornerstone of Large Language Model (LLM) development, predicts improvements in model performance with increasing computational resources.
arXiv AI
Jul 28
Scale Weight Decay and Train Better
arXiv:2607. 23777v1 Announce Type: cross Abstract: The discovery of scaling laws has motivated training neural networks on ever increasing quantities of data.