arXiv AI
Jul 8
Transformers converge to invariant algorithmic cores
arXiv:2602. 22600v2 Announce Type: replace-cross Abstract: Training selects for behavior, not circuitry: many weight configurations can implement the same function.
Latent Algorithmic Structure Precedes Grokking: A Mechanistic Study of ReLU MLPs on Modular Arithmetic
Related stories
arXiv Machine Learning
Sep 23
A Spectral Theory of Grokking: Weight Decay induces Feature Learning
The paper presents a spectral theory explaining the phenomenon of grokking, where an initial fit to training data is followed by a delayed improvement in generalization. It shows that for homogeneous networks trained with squared loss and L₂ weight decay, residuals after memorization influence the neural tangent kernel (NTK) dynamics, leading to a transition from lazy to rich learning. The theory predicts that grokking timescales depend on the product of learning rate and weight decay, and that stronger decay can halt fitting, with empirical validation on modular addition tasks using MLPs and Transformers.
arXiv Machine Learning
Aug 3
The Grokked Illusion: True Equilibrium Mitigates Catastrophic Forgetting
arXiv:2607. 29503v1 Announce Type: new Abstract: While neural networks are typically evaluated by their training and test performance, these metrics do not reveal how robust a learned representation is.
arXiv AI
Jun 17
The Discrete-Log Clock: How a Transformer Learns Modular Multiplication
arXiv:2606. 17399v1 Announce Type: cross Abstract: When small transformers grok modular multiplication, prior work reports that the learned embedding has a "dense" Fourier spectrum requiring all frequencies.
arXiv Machine Learning
Oct 7
CNet: A Complex-Valued Deep Learning Framework with Wirtinger Autodifferentiation and FFT--Hadamard Convolution
arXiv:2610.08592v1 Announce Type: new Abstract: CNet is a C++/CUDA framework for building and training deep complex-valued neural networks (CVNNs) and, more generally, for optimizing complex-valued f...
arXiv AI
Oct 2
Misalignment of Low-Loss Regions Causes Grokking
arXiv:2610.00620v1 Announce Type: cross Abstract: Grokking refers to the delayed emergence of validation-set generalization after a model has already overfit the training set. Although first observed...
arXiv Machine Learning
Jul 1
A Theory of How Pretraining Shapes Inductive Bias in Fine-Tuning
arXiv:2602. 20062v2 Announce Type: replace Abstract: Pretraining and fine-tuning are central stages in modern machine learning systems.
arXiv Machine Learning
Sep 11
Quantifying the Memorization-to-Generalization Transition: Scaling Laws and Phase Structure in Grokking
The study investigates the delayed transition from memorization to generalization—known as grokking—in two‑hidden‑layer MLPs trained on modular arithmetic. By exploring 384 hyperparameter configurations, the authors derive a power‑law scaling relation for the onset time of generalization, showing that data complexity dominates over model capacity. A clear phase boundary at weight decay around 1.0 separates grokking from non‑grokking regimes, and weight norm trajectories indicate implicit regularization during the transition.
arXiv AI
6d ago
Subspace Crystallization in Grokking: Rank Compression Tracks the Phase Transition
The paper investigates the relationship between rank compression of weight matrices and the phenomenon of grokking in neural networks. Using a mean‑field MLP and a one‑layer transformer trained on modular arithmetic, the authors find that rank compression occurs after the onset of grokking, not during memorization, and that the effective rank drops sharply around 900 steps following the grokking onset. They also observe a subspace crystallization effect, where the number of singular directions needed to preserve test accuracy shrinks dramatically during the transition, and demonstrate that artificially imposing low rank does not accelerate grokking, indicating that compression is a signature rather than a cause of the phenomenon.
arXiv AI
Oct 9
Probability-Signature Dynamics: Unpacking Modular Addition Learning Within Two-Layer Networks
The paper investigates why neural networks trained on modular addition tasks develop Fourier-structured representations that enable exact generalization. Using probability signatures, the authors show that leading gradient interactions are cyclic shift operators, which are diagonalized by the discrete Fourier transform, leading to decoupled Fourier-mode dynamics that explain Fourier sparsity, frequency matching, and phase alignment. The framework also clarifies how label noise can accelerate early loss decrease by increasing conditional label collisions, and it is demonstrated to predict frequency behavior in other operators such as XOR.
arXiv Machine Learning
Jun 5
Deciphering Two Training Clocks in Grokking via Deep Linear Network Theory with Conditional ReLU Reduction
arXiv:2606. 05863v1 Announce Type: new Abstract: Grokking suggests that fitting the training data and learning a simple underlying rule may occur on different time scales.
arXiv Machine Learning
Oct 22
A Mechanistic Study of Algorithmic Generalization in Diffusion Models on Modular Arithmetic
The paper investigates how diffusion models learn and generalize algorithmically, focusing on a modular addition task with images. In a single‑image setting, the model encodes operands as periodic representations and combines them via self‑attention and feed‑forward layers that mirror trigonometric addition identities. In a diverse‑image setting, the sampling process separates into an arithmetic computation phase and a visual refinement phase, with the transition marked by an internal frequency‑entropy signal.