arXiv Machine Learning September 23, 2026 By Lenz Pracher, Pascal de Jong, Oskar Lieshaus, Alan Jeffares, Steffen Rulands

A Spectral Theory of Grokking: Weight Decay induces Feature Learning

Read the original on arXiv Machine Learning →

The paper presents a spectral theory explaining the phenomenon of grokking, where an initial fit to training data is followed by a delayed improvement in generalization. It shows that for homogeneous networks trained with squared loss and L₂ weight decay, residuals after memorization influence the neural tangent kernel (NTK) dynamics, leading to a transition from lazy to rich learning. The theory predicts that grokking timescales depend on the product of learning rate and weight decay, and that stronger decay can halt fitting, with empirical validation on modular addition tasks using MLPs and Transformers.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

llms

arXiv AI

Jul 28

Grokking on the Weight-Decay Clock: A Rate Hierarchy from Softly Broken Symmetries

arXiv:2607. 23967v1 Announce Type: new Abstract: Delayed generalization, or grokking, remains poorly understood despite extensive empirical study.

By Taeyoung Kim

arXiv Machine Learning

Sep 11

Quantifying the Memorization-to-Generalization Transition: Scaling Laws and Phase Structure in Grokking

The study investigates the delayed transition from memorization to generalization—known as grokking—in two‑hidden‑layer MLPs trained on modular arithmetic. By exploring 384 hyperparameter configurations, the authors derive a power‑law scaling relation for the onset time of generalization, showing that data complexity dominates over model capacity. A clear phase boundary at weight decay around 1.0 separates grokking from non‑grokking regimes, and weight norm trajectories indicate implicit regularization during the transition.

By Anish Kataria

arXiv Machine Learning

Sep 15

Optimal Learning Rate Schedules under Functional Scaling Laws: Power Decay and Warmup-Stable-Decay

arXiv:2602.06797v3 Announce Type: replace-cross Abstract: We study optimal learning rate (LR) schedules under the functional scaling law (FSL) framework (Li et al., 2025), which decomposes training d...

By Binghui Li, Zilin Wang, Fengling Chen, Shiyang Zhao, Ruiheng Zheng, Lei Wu

arXiv Machine Learning

Jun 5

Deciphering Two Training Clocks in Grokking via Deep Linear Network Theory with Conditional ReLU Reduction

arXiv:2606. 05863v1 Announce Type: new Abstract: Grokking suggests that fitting the training data and learning a simple underlying rule may occur on different time scales.

By Hu Tan, Kuo Gai, Shihua Zhang

rag

arXiv AI

Jun 2

The Geometry of Grokking: Norm Minimization on the Zero-Loss Manifold

arXiv:2511. 01938v3 Announce Type: replace-cross Abstract: Grokking is a puzzling phenomenon in neural networks where full generalization occurs only after a substantial delay following the complete memorization of the training data.

By Tiberiu Musat

arXiv Machine Learning

Jul 7

Structure-Specific Representational Priors Causally Control the Grokking Delay

arXiv:2607. 04333v1 Announce Type: new Abstract: Grokking -- generalization arriving long after training-set interpolation -- can be accelerated by structure-agnostic interventions: gradient filtering, weight-norm clamping, geometric penalties on hidden representations.

By Gunner Levi Howe

llms

One email a morning, machine-written

Related stories

Grokking on the Weight-Decay Clock: A Rate Hierarchy from Softly Broken Symmetries

Quantifying the Memorization-to-Generalization Transition: Scaling Laws and Phase Structure in Grokking

Optimal Learning Rate Schedules under Functional Scaling Laws: Power Decay and Warmup-Stable-Decay

Deciphering Two Training Clocks in Grokking via Deep Linear Network Theory with Conditional ReLU Reduction

The Geometry of Grokking: Norm Minimization on the Zero-Loss Manifold

Structure-Specific Representational Priors Causally Control the Grokking Delay