← Back to all news
arXiv Machine Learning

Quantifying the Memorization-to-Generalization Transition: Scaling Laws and Phase Structure in Grokking

The study investigates the delayed transition from memorization to generalization—known as grokking—in two‑hidden‑layer MLPs trained on modular arithmetic. By exploring 384 hyperparameter configurations, the authors derive a power‑law scaling relation for the onset time of generalization, showing that data complexity dominates over model capacity. A clear phase boundary at weight decay around 1.0 separates grokking from non‑grokking regimes, and weight norm trajectories indicate implicit regularization during the transition.

Related stories

arXiv Machine Learning
Sep 23

A Spectral Theory of Grokking: Weight Decay induces Feature Learning

The paper presents a spectral theory explaining the phenomenon of grokking, where an initial fit to training data is followed by a delayed improvement in generalization. It shows that for homogeneous networks trained with squared loss and L₂ weight decay, residuals after memorization influence the neural tangent kernel (NTK) dynamics, leading to a transition from lazy to rich learning. The theory predicts that grokking timescales depend on the product of learning rate and weight decay, and that stronger decay can halt fitting, with empirical validation on modular addition tasks using MLPs and Transformers.

By Lenz Pracher, Pascal de Jong, Oskar Lieshaus, Alan Jeffares, Steffen Rulands
llms
More like this →
arXiv Machine Learning
Oct 19

Statistical Vignettes on the Grokking Phenomenon

The paper investigates grokking in a simple, analytically tractable setting, exploring how initialization, signal strength, weight decay, sample size, and input dimension influence learning curves. It distinguishes between early fitting via nuisance features, core feature learning, and delayed generalization, providing sharp conditions for provable grokking and analytic expressions for generalization plateau lengths. The study also characterizes regimes of early generalization, failure to reach target accuracy, and identifies the interpolation edge as a distinct dynamical regime where vanishing spectral gaps separate core learning from generalization.

By Baye Malick Gning, Elvis Dohmatob
More like this →
arXiv AI
Jun 30

A Stochastic--Geometric Theory of Scaling Laws in Grokking

arXiv:2606. 30388v1 Announce Type: cross Abstract: Delayed generalization (\ie~grokking) refers to the phenomenon in which a neural network fits its training data early in training but only begins to generalize after a prolonged delay, often through an abrupt transition.

By R\'ois\'in Luo, Christian Gagn\'e, Jonas Ngnaw\'e, Ihsan Ullah, Karyn Morrissey
More like this →
arXiv AI
5d ago

Scaling and Renormalization in Neural Networks: An Effective Renormalization-Group Description of Grokking

The paper investigates the phenomenon of grokking—a delayed shift from memorization to generalization—in two‑layer neural networks. By treating an ensemble of trained models as a statistical system, the authors define an order parameter (squared weight norm) and use heat‑capacity‑ and susceptibility‑like proxies to map training epoch and weight decay to thermal and field controls. They construct an exact coupling matrix via a second‑order loss expansion, employ its Fiedler vector to impose an effective geometry, and perform real‑space renormalization‑group (RG) transformations through block‑spin aggregation. Linearizing the resulting RG map yields scaling exponents that agree with empirical estimates within 1–10%, providing a quantitative physical framework for neural generalization transitions.

By Min Chen, Junqi Wang, Zeguan Wu, Wenhao Huang, Junyu Liu
More like this →