The paper investigates the phenomenon of grokking—a delayed shift from memorization to generalization—in two‑layer neural networks. By treating an ensemble of trained models as a statistical system, the authors define an order parameter (squared weight norm) and use heat‑capacity‑ and susceptibility‑like proxies to map training epoch and weight decay to thermal and field controls. They construct an exact coupling matrix via a second‑order loss expansion, employ its Fiedler vector to impose an effective geometry, and perform real‑space renormalization‑group (RG) transformations through block‑spin aggregation. Linearizing the resulting RG map yields scaling exponents that agree with empirical estimates within 1–10%, providing a quantitative physical framework for neural generalization transitions.
arXiv:2608. 01833v1 Announce Type: cross Abstract: Grokking is a striking phenomenon in neural network training, where a model can undergo a prolonged period of pure memorization before abrupt generalization.
arXiv:2511. 07308v3 Announce Type: replace Abstract: Understanding the training dynamics of deep neural networks remains a major open problem, with physics-inspired approaches offering promising insights.
The study investigates the delayed transition from memorization to generalization—known as grokking—in two‑hidden‑layer MLPs trained on modular arithmetic. By exploring 384 hyperparameter configurations, the authors derive a power‑law scaling relation for the onset time of generalization, showing that data complexity dominates over model capacity. A clear phase boundary at weight decay around 1.0 separates grokking from non‑grokking regimes, and weight norm trajectories indicate implicit regularization during the transition.
arXiv:2501. 02436v5 Announce Type: replace Abstract: Advancements in artificial intelligence call for a deeper understanding of the fundamental mechanisms underlying deep learning.
The paper introduces a supervised, scale‑shared neural architecture for two‑dimensional site percolation, implementing a neural renormalization group flow. The model recursively applies a learned coarse‑graining rule across scales, producing a latent field that predicts crossing probability and a fine‑graining decoder that reconstructs the largest‑cluster mask. Trained only on small lattices, it extrapolates to larger systems, accurately recovers the spanning cluster, and yields observables that follow expected finite‑size scaling near the critical point, highlighting the importance of critical fluctuations in the latent representation.
The paper presents a supervised, scale‑shared neural architecture that learns a coarse‑graining rule for two‑dimensional site percolation. By recursively applying this rule, the model generates a latent field from which the crossing probability is predicted and a fine‑graining decoder reconstructs the largest‑cluster mask. Trained only on small lattices, the network extrapolates to larger systems, accurately recovers the spanning cluster, and reproduces finite‑size scaling near the critical point, demonstrating that the latent representation captures critical fluctuations and scale‑dependent flows consistent with renormalization‑group theory.
arXiv:2607. 07127v1 Announce Type: cross Abstract: Lattice field theory is the workhorse of non-perturbative physics, used to simulate phenomena from the strong nuclear force to critical phenomena in materials.
arXiv:2608. 19331v1 Announce Type: cross Abstract: We modify the NN/QFT duality [1] to incorporate the layerwise permutation symmetry of the network, resulting in a $(0\!
arXiv:2608. 06597v1 Announce Type: cross Abstract: A scientific theory of deep learning, comprising learning dynamics and statistical properties of learned models, is rapidly gaining attention.
arXiv:2606. 10324v1 Announce Type: new Abstract: The analogy between deep neural network forward passes and renormalization group (RG) flows has been repeatedly noted in the literature, but existing treatments remain qualitative: depth is described as a coarse-graining scale, attention is likened to a partition function, and representations are said to flow toward fixed points.
arXiv:2606. 00157v1 Announce Type: cross Abstract: We consider establishing the interpretability theory of deep learning through constructing a corresponding relationship between the renormalization group (RG) method in statistical physics and the training process of deep neural networks (DNNs).
arXiv:2508. 11522v4 Announce Type: replace Abstract: Neural tangent kernels (NTKs) are a powerful tool for analyzing deep, non-linear neural networks.