← Back to all news
arXiv Machine Learning By Anish Kataria

Quantifying the Memorization-to-Generalization Transition: Scaling Laws and Phase Structure in Grokking

Read the original on arXiv Machine Learning →

The study investigates the delayed transition from memorization to generalization—known as grokking—in two‑hidden‑layer MLPs trained on modular arithmetic. By exploring 384 hyperparameter configurations, the authors derive a power‑law scaling relation for the onset time of generalization, showing that data complexity dominates over model capacity. A clear phase boundary at weight decay around 1.0 separates grokking from non‑grokking regimes, and weight norm trajectories indicate implicit regularization during the transition.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Sep 23

A Spectral Theory of Grokking: Weight Decay induces Feature Learning

The paper presents a spectral theory explaining the phenomenon of grokking, where an initial fit to training data is followed by a delayed improvement in generalization. It shows that for homogeneous networks trained with squared loss and L₂ weight decay, residuals after memorization influence the neural tangent kernel (NTK) dynamics, leading to a transition from lazy to rich learning. The theory predicts that grokking timescales depend on the product of learning rate and weight decay, and that stronger decay can halt fitting, with empirical validation on modular addition tasks using MLPs and Transformers.

By Lenz Pracher, Pascal de Jong, Oskar Lieshaus, Alan Jeffares, Steffen Rulands
llms
More like this →
arXiv Machine Learning
Oct 19

Statistical Vignettes on the Grokking Phenomenon

The paper investigates grokking in a simple, analytically tractable setting, exploring how initialization, signal strength, weight decay, sample size, and input dimension influence learning curves. It distinguishes between early fitting via nuisance features, core feature learning, and delayed generalization, providing sharp conditions for provable grokking and analytic expressions for generalization plateau lengths. The study also characterizes regimes of early generalization, failure to reach target accuracy, and identifies the interpolation edge as a distinct dynamical regime where vanishing spectral gaps separate core learning from generalization.

By Baye Malick Gning, Elvis Dohmatob
More like this →