The paper investigates how overparameterized neural networks can fit training data in many different ways yet generalize differently. By applying short, fixed‑duration weight‑decay pulses during the grokking plateau, the authors show that early in training the pulses produce unordered shifts in generalization time, but later a clear dose ordering emerges: larger weight‑decay increases lead to earlier generalization, while larger decreases delay it. This ordering appears before visible generalization and persists even as test‑loss barriers collapse, a phenomenon the authors term the canalization of function selection.
By Yiming Lin
For overparameterized neural networks, many solutions can fit the training data equally well while behaving very differently on unseen samples. Grokking separates training fit from visible generalizat...
arXiv:2607. 23967v1 Announce Type: new Abstract: Delayed generalization, or grokking, remains poorly understood despite extensive empirical study.
By Taeyoung Kim
The study investigates the delayed transition from memorization to generalization—known as grokking—in two‑hidden‑layer MLPs trained on modular arithmetic. By exploring 384 hyperparameter configurations, the authors derive a power‑law scaling relation for the onset time of generalization, showing that data complexity dominates over model capacity. A clear phase boundary at weight decay around 1.0 separates grokking from non‑grokking regimes, and weight norm trajectories indicate implicit regularization during the transition.
By Anish Kataria
arXiv:2606. 30388v1 Announce Type: cross Abstract: Delayed generalization (\ie~grokking) refers to the phenomenon in which a neural network fits its training data early in training but only begins to generalize after a prolonged delay, often through an abrupt transition.
By R\'ois\'in Luo, Christian Gagn\'e, Jonas Ngnaw\'e, Ihsan Ullah, Karyn Morrissey
The paper investigates whether flatter loss landscapes drive better generalization in the context of Grokking. It shows that sharpness‑aware minimization (SAM) alone does not reliably induce the transition to generalization, but when combined with weight decay, SAM can accelerate this transition up to four times faster. The authors provide a theoretical analysis using a minimal two‑layer ReLU model, demonstrating that flatness alone cannot distinguish memorizing from generalizing solutions, while weight decay favors generalization and a local stability window explains SAM’s effect.
By Mohnish Harwani
The paper presents a spectral theory explaining the phenomenon of grokking, where an initial fit to training data is followed by a delayed improvement in generalization. It shows that for homogeneous networks trained with squared loss and L₂ weight decay, residuals after memorization influence the neural tangent kernel (NTK) dynamics, leading to a transition from lazy to rich learning. The theory predicts that grokking timescales depend on the product of learning rate and weight decay, and that stronger decay can halt fitting, with empirical validation on modular addition tasks using MLPs and Transformers.
By Lenz Pracher, Pascal de Jong, Oskar Lieshaus, Alan Jeffares, Steffen Rulands
The paper investigates grokking in a simple, analytically tractable setting, exploring how initialization, signal strength, weight decay, sample size, and input dimension influence learning curves. It distinguishes between early fitting via nuisance features, core feature learning, and delayed generalization, providing sharp conditions for provable grokking and analytic expressions for generalization plateau lengths. The study also characterizes regimes of early generalization, failure to reach target accuracy, and identifies the interpolation edge as a distinct dynamical regime where vanishing spectral gaps separate core learning from generalization.
By Baye Malick Gning, Elvis Dohmatob
arXiv:2601. 19791v4 Announce Type: replace Abstract: We study grokking, the onset of generalization long after overfitting, in a classical ridge regression setting.
By Mingyue Xu, Gal Vardi, Itay Safran
arXiv:2607. 11666v1 Announce Type: new Abstract: Grokking is a phenomenon in which neural networks initially memorize training data and only later exhibit strong generalization after prolonged optimization.
By Maksim A Kazanskii
arXiv:2609.07755v1 Announce Type: new
Abstract: Understanding generalization remains a central challenge in machine learning because it requires jointly considering data, architecture, and training d...
By Yuqing Wang, Ioannis G. Kevrekidis, Mikhail Belkin
The paper derives an exact discrete‑time law that captures how learning‑rate schedules and weight decay interact in scale‑invariant neural networks, showing that a single scalar quantity governs the effective step size. It demonstrates that the balance point between contraction and expansion is intrinsically unstable, leading to recurrent dynamics when using constant learning rates with weight decay. The authors extend this analysis to various optimizers and datasets, confirming the law’s precision and showing that performance peaks sharply at the predicted boundary.
By Hasan Amin, Wei-Kai Chang, Rajiv Khanna