arXiv AI
Jul 28
Grokking on the Weight-Decay Clock: A Rate Hierarchy from Softly Broken Symmetries
arXiv:2607. 23967v1 Announce Type: new Abstract: Delayed generalization, or grokking, remains poorly understood despite extensive empirical study.
Quantifying the Memorization-to-Generalization Transition: Scaling Laws and Phase Structure in Grokking
The study investigates the delayed transition from memorization to generalization—known as grokking—in two‑hidden‑layer MLPs trained on modular arithmetic. By exploring 384 hyperparameter configurations, the authors derive a power‑law scaling relation for the onset time of generalization, showing that data complexity dominates over model capacity. A clear phase boundary at weight decay around 1.0 separates grokking from non‑grokking regimes, and weight norm trajectories indicate implicit regularization during the transition.
Related stories
arXiv AI
Jul 9
At-Grok Is Not Converged:A Measurement-Validity Audit for Grokking Representation Metrics
arXiv:2607. 06639v1 Announce Type: cross Abstract: On modular arithmetic, a network's embedding keeps compressing for tens of thousands of steps after it has already generalized.
arXiv Machine Learning
Aug 4
Tunneling the Loss Landscape: Bypassing Memorization with Monte Carlo Parameter Swapping
arXiv:2608. 01833v1 Announce Type: cross Abstract: Grokking is a striking phenomenon in neural network training, where a model can undergo a prolonged period of pure memorization before abrupt generalization.
arXiv AI
Jun 15
The Weight Norm Sets the Grokking Timescale: A Causal Delay Law
arXiv:2606. 13753v1 Announce Type: cross Abstract: Grokking is the delayed onset of generalization in neural networks, arising long after they fit the training data.
arXiv AI
Jun 30
A Stochastic--Geometric Theory of Scaling Laws in Grokking
arXiv:2606. 30388v1 Announce Type: cross Abstract: Delayed generalization (\ie~grokking) refers to the phenomenon in which a neural network fits its training data early in training but only begins to generalize after a prolonged delay, often through an abrupt transition.
arXiv AI
Jun 9
Emergence via Phase Transitions: Mechanism Landscapes and Universal Convergence Across Complex Systems
arXiv:2606. 07563v1 Announce Type: cross Abstract: Across machine learning, biology, and physics, independently evolving systems often converge toward strikingly similar high-level structures despite radically different microscopic details.
arXiv Machine Learning
Jun 17
Noise-Driven Escape from Metastable Phases explains Grokking in Deep Neural Networks
arXiv:2606. 17120v1 Announce Type: new Abstract: Deep neural networks (DNNs) exhibit first order phase transitions under variations of the L2 regularization strength, with each transition marking the onset of a new learnable feature.
arXiv Machine Learning
Jul 24
Thermodynamic Weight Decay: Exploring Grokking Acceleration via Attention Specific Heat
arXiv:2607. 20552v1 Announce Type: new Abstract: Grokking -- the delayed generalization of neural networks long after they have memorized their training data -- wastes thousands of training epochs and is notoriously unpredictable.
arXiv AI
Jul 7
Grokking Is Conditional and Fragile: A Fully-Tractable, Multi-Seed Study at 12K Parameters
arXiv:2607. 05104v1 Announce Type: cross Abstract: Grokking -- the delayed onset of generalization long after a network has fit its training set - -is usually studied in models too large to read completely and reported from single training runs.
arXiv Machine Learning
Aug 14
Neural Quadratic Forms: A Unified Minimal Model for Sudden Learning and Scaling Laws
arXiv:2608. 13335v1 Announce Type: new Abstract: Neural networks trained by gradient descent on a smooth cost function can nevertheless learn in steps: the cost holds on long plateaus and then drops abruptly.
arXiv Machine Learning
Jun 30
Anti-Collapse Dynamics and the Emergence of Multi-Time-Scale Learning in Recurrent Neural Networks
arXiv:2606. 29519v1 Announce Type: new Abstract: Long-range learning is hard for recurrent networks trained with stochastic gradient descent, because the influence of a past input fades with the lag $\ell$, and if it fades too fast the dependence cannot be learned from finite data.
Hugging Face Trending Papers
Jun 11
Circuit Synchronization Precedes Generalization: Causal Evidence from Fourier Structure in Grokking Transformers
Grokking -- where a transformer on modular arithmetic suddenly transitions from near-chance to near-perfect validation accuracy -- is attributed to a Fourier circuit, but its timing, causal structure, and controllability remain poorly understood. We introduce the Frequency Synchronization Degree (FSD), a normalised, permutation-tested metric for Fourier circuit synchronisation requiring no prior circuit knowledge.