arXiv AI
Latent Algorithmic Structure Precedes Grokking: A Mechanistic Study of ReLU MLPs on Modular Arithmetic
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Jul 8
Transformers converge to invariant algorithmic cores
arXiv:2602. 22600v2 Announce Type: replace-cross Abstract: Training selects for behavior, not circuitry: many weight configurations can implement the same function.
arXiv Machine Learning
Sep 23
A Spectral Theory of Grokking: Weight Decay induces Feature Learning
The paper presents a spectral theory explaining the phenomenon of grokking, where an initial fit to training data is followed by a delayed improvement in generalization. It shows that for homogeneous networks trained with squared loss and L₂ weight decay, residuals after memorization influence the neural tangent kernel (NTK) dynamics, leading to a transition from lazy to rich learning. The theory predicts that grokking timescales depend on the product of learning rate and weight decay, and that stronger decay can halt fitting, with empirical validation on modular addition tasks using MLPs and Transformers.
arXiv Machine Learning
Aug 3
The Grokked Illusion: True Equilibrium Mitigates Catastrophic Forgetting
arXiv:2607. 29503v1 Announce Type: new Abstract: While neural networks are typically evaluated by their training and test performance, these metrics do not reveal how robust a learned representation is.
arXiv AI
Jun 17
The Discrete-Log Clock: How a Transformer Learns Modular Multiplication
arXiv:2606. 17399v1 Announce Type: cross Abstract: When small transformers grok modular multiplication, prior work reports that the learned embedding has a "dense" Fourier spectrum requiring all frequencies.
arXiv Machine Learning
Oct 7
CNet: A Complex-Valued Deep Learning Framework with Wirtinger Autodifferentiation and FFT--Hadamard Convolution
arXiv:2610.08592v1 Announce Type: new Abstract: CNet is a C++/CUDA framework for building and training deep complex-valued neural networks (CVNNs) and, more generally, for optimizing complex-valued f...
arXiv AI
Oct 2
Misalignment of Low-Loss Regions Causes Grokking
arXiv:2610.00620v1 Announce Type: cross Abstract: Grokking refers to the delayed emergence of validation-set generalization after a model has already overfit the training set. Although first observed...