arXiv:2607. 20552v1 Announce Type: new Abstract: Grokking -- the delayed generalization of neural networks long after they have memorized their training data -- wastes thousands of training epochs and is notoriously unpredictable.
By Chitraansh Pandey
arXiv:2607. 06639v1 Announce Type: cross Abstract: On modular arithmetic, a network's embedding keeps compressing for tens of thousands of steps after it has already generalized.
By Truong Xuan Khanh
The paper introduces the fully‑enumerable transformer—a tiny transformer trained on tasks where every input can be evaluated exactly—as a scientific instrument for studying delayed generalization. It claims four unique capabilities: exact, falsifiable generalization ceilings; precise task surgery; direct observation of all weights; and survival‑time statistics that treat non‑grokking as censored data. A preregistered conservation study shows that two task‑side laws (a recoverability‑ceiling law and a role‑conflict delay law) hold across a 4,000‑fold increase in model size, while a weight‑decay law deforms predictably, demonstrating that the instrument can reveal lawful patterns of generalization across scales.
By Yoshiyuki Ootani
arXiv:2608. 07436v1 Announce Type: new Abstract: Under the standard split, Muon gets hidden matrices and AdamW embeddings/output head.
By Ali Janati, Kaoutar El Maghraoui, Andrei Kanavalau, Anass Belfatmi
The study investigates the delayed transition from memorization to generalization—known as grokking—in two‑hidden‑layer MLPs trained on modular arithmetic. By exploring 384 hyperparameter configurations, the authors derive a power‑law scaling relation for the onset time of generalization, showing that data complexity dominates over model capacity. A clear phase boundary at weight decay around 1.0 separates grokking from non‑grokking regimes, and weight norm trajectories indicate implicit regularization during the transition.
By Anish Kataria
Grokking -- where a transformer on modular arithmetic suddenly transitions from near-chance to near-perfect validation accuracy -- is attributed to a Fourier circuit, but its timing, causal structure, and controllability remain poorly understood. We introduce the Frequency Synchronization Degree (FSD), a normalised, permutation-tested metric for Fourier circuit synchronisation requiring no prior circuit knowledge.