arXiv AI
Jun 15
The Weight Norm Sets the Grokking Timescale: A Causal Delay Law
arXiv:2606. 13753v1 Announce Type: cross Abstract: Grokking is the delayed onset of generalization in neural networks, arising long after they fit the training data.
What Does the Weight Norm Control in Grokking? Logit-Scale Mediation under Cross-Entropy
arXiv:2606. 18465v1 Announce Type: cross Abstract: Grokking, the delayed jump from memorization to generalization, is usually tied to the weight norm: a smaller norm generalizes sooner.
Related stories
arXiv AI
Jul 9
At-Grok Is Not Converged:A Measurement-Validity Audit for Grokking Representation Metrics
arXiv:2607. 06639v1 Announce Type: cross Abstract: On modular arithmetic, a network's embedding keeps compressing for tens of thousands of steps after it has already generalized.
arXiv Machine Learning
Jul 24
Thermodynamic Weight Decay: Exploring Grokking Acceleration via Attention Specific Heat
arXiv:2607. 20552v1 Announce Type: new Abstract: Grokking -- the delayed generalization of neural networks long after they have memorized their training data -- wastes thousands of training epochs and is notoriously unpredictable.
arXiv AI
Jul 7
Grokking Is Conditional and Fragile: A Fully-Tractable, Multi-Seed Study at 12K Parameters
arXiv:2607. 05104v1 Announce Type: cross Abstract: Grokking -- the delayed onset of generalization long after a network has fit its training set - -is usually studied in models too large to read completely and reported from single training runs.
arXiv AI
Jul 28
Grokking on the Weight-Decay Clock: A Rate Hierarchy from Softly Broken Symmetries
arXiv:2607. 23967v1 Announce Type: new Abstract: Delayed generalization, or grokking, remains poorly understood despite extensive empirical study.
arXiv AI
Jul 24
The Active Ingredient in Muon's Grokking
arXiv:2607. 20512v1 Announce Type: cross Abstract: The Muon optimizer reaches the grokking threshold on modular arithmetic faster than AdamW.
arXiv AI
Aug 10
Post-Grokking Collapse at the Representation-Readout Interface in Muon-Trained Transformers
arXiv:2608. 07436v1 Announce Type: new Abstract: Under the standard split, Muon gets hidden matrices and AdamW embeddings/output head.
arXiv Machine Learning
Jun 5
Deciphering Two Training Clocks in Grokking via Deep Linear Network Theory with Conditional ReLU Reduction
arXiv:2606. 05863v1 Announce Type: new Abstract: Grokking suggests that fitting the training data and learning a simple underlying rule may occur on different time scales.
arXiv AI
Jun 4
Low-Rank Decay for Grokking in Scale-Invariant Transformers: A Spectral-Geometric View
arXiv:2606. 04405v1 Announce Type: cross Abstract: Modern Transformer architectures frequently employ normalization mechanisms such as RMSNorm and Query-Key Normalization, making parts of the model approximately scale-invariant with respect to weight magnitudes.
arXiv Machine Learning
Jul 7
Structure-Specific Representational Priors Causally Control the Grokking Delay
arXiv:2607. 04333v1 Announce Type: new Abstract: Grokking -- generalization arriving long after training-set interpolation -- can be accelerated by structure-agnostic interventions: gradient filtering, weight-norm clamping, geometric penalties on hidden representations.
arXiv Machine Learning
Jun 19
Weibull Weight-Scale Parameter Evolution under AdamW Training Dynamics
arXiv:2606. 19367v1 Announce Type: new Abstract: Building on a two-parameter Weibull framework for diagnosing transformer weight distributions, we study why the Weibull weight-scale parameter $\lambda$ grows, overshoots, and then relaxes during AdamW training.
arXiv Machine Learning
Jul 14
Gauge dependence and structured-output corruption in sign-branched repetition penalties: measurements across models, inference stacks, and alternative repetition controls
arXiv:2607. 09791v1 Announce Type: new Abstract: The multiplicative repetition penalty shipped across the LLM inference ecosystem (HuggingFace, vLLM, llama.