arXiv Machine Learning
Mapping the Emergence of Regularization-Driven Dynamics in Grokking
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Aug 27
Canalization Before Generalization: Grokking as a Dynamical Probe
The paper investigates how overparameterized neural networks can fit training data in many different ways yet generalize differently. By applying short, fixed‑duration weight‑decay pulses during the grokking plateau, the authors show that early in training the pulses produce unordered shifts in generalization time, but later a clear dose ordering emerges: larger weight‑decay increases lead to earlier generalization, while larger decreases delay it. This ordering appears before visible generalization and persists even as test‑loss barriers collapse, a phenomenon the authors term the canalization of function selection.
Hugging Face Trending Papers
Aug 26
Canalization Before Generalization: Grokking as a Dynamical Probe
For overparameterized neural networks, many solutions can fit the training data equally well while behaving very differently on unseen samples. Grokking separates training fit from visible generalizat...
arXiv AI
Jul 28
Grokking on the Weight-Decay Clock: A Rate Hierarchy from Softly Broken Symmetries
arXiv:2607. 23967v1 Announce Type: new Abstract: Delayed generalization, or grokking, remains poorly understood despite extensive empirical study.
arXiv Machine Learning
Sep 11
Quantifying the Memorization-to-Generalization Transition: Scaling Laws and Phase Structure in Grokking
The study investigates the delayed transition from memorization to generalization—known as grokking—in two‑hidden‑layer MLPs trained on modular arithmetic. By exploring 384 hyperparameter configurations, the authors derive a power‑law scaling relation for the onset time of generalization, showing that data complexity dominates over model capacity. A clear phase boundary at weight decay around 1.0 separates grokking from non‑grokking regimes, and weight norm trajectories indicate implicit regularization during the transition.
arXiv AI
Jun 30
A Stochastic--Geometric Theory of Scaling Laws in Grokking
arXiv:2606. 30388v1 Announce Type: cross Abstract: Delayed generalization (\ie~grokking) refers to the phenomenon in which a neural network fits its training data early in training but only begins to generalize after a prolonged delay, often through an abrupt transition.
arXiv Machine Learning
Oct 9
Do Flatter Minima Drive Better Generalization? An Algorithmic Separation in Grokking
The paper investigates whether flatter loss landscapes drive better generalization in the context of Grokking. It shows that sharpness‑aware minimization (SAM) alone does not reliably induce the transition to generalization, but when combined with weight decay, SAM can accelerate this transition up to four times faster. The authors provide a theoretical analysis using a minimal two‑layer ReLU model, demonstrating that flatness alone cannot distinguish memorizing from generalizing solutions, while weight decay favors generalization and a local stability window explains SAM’s effect.