← Back to all news
arXiv AI By Min Chen, Junqi Wang, Zeguan Wu, Wenhao Huang, Junyu Liu

Scaling and Renormalization in Neural Networks: An Effective Renormalization-Group Description of Grokking

Read the original on arXiv AI →

The paper investigates the phenomenon of grokking—a delayed shift from memorization to generalization—in two‑layer neural networks. By treating an ensemble of trained models as a statistical system, the authors define an order parameter (squared weight norm) and use heat‑capacity‑ and susceptibility‑like proxies to map training epoch and weight decay to thermal and field controls. They construct an exact coupling matrix via a second‑order loss expansion, employ its Fiedler vector to impose an effective geometry, and perform real‑space renormalization‑group (RG) transformations through block‑spin aggregation. Linearizing the resulting RG map yields scaling exponents that agree with empirical estimates within 1–10%, providing a quantitative physical framework for neural generalization transitions.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Machine Learning
Sep 11

Quantifying the Memorization-to-Generalization Transition: Scaling Laws and Phase Structure in Grokking

The study investigates the delayed transition from memorization to generalization—known as grokking—in two‑hidden‑layer MLPs trained on modular arithmetic. By exploring 384 hyperparameter configurations, the authors derive a power‑law scaling relation for the onset time of generalization, showing that data complexity dominates over model capacity. A clear phase boundary at weight decay around 1.0 separates grokking from non‑grokking regimes, and weight norm trajectories indicate implicit regularization during the transition.

By Anish Kataria
More like this →
Hugging Face Trending Papers
Aug 27

Neural Renormalization Group Flow for Percolation

The paper introduces a supervised, scale‑shared neural architecture for two‑dimensional site percolation, implementing a neural renormalization group flow. The model recursively applies a learned coarse‑graining rule across scales, producing a latent field that predicts crossing probability and a fine‑graining decoder that reconstructs the largest‑cluster mask. Trained only on small lattices, it extrapolates to larger systems, accurately recovers the spanning cluster, and yields observables that follow expected finite‑size scaling near the critical point, highlighting the importance of critical fluctuations in the latent representation.

More like this →
arXiv Machine Learning
Aug 28

Neural Renormalization Group Flow for Percolation

The paper presents a supervised, scale‑shared neural architecture that learns a coarse‑graining rule for two‑dimensional site percolation. By recursively applying this rule, the model generates a latent field from which the crossing probability is predicted and a fine‑graining decoder reconstructs the largest‑cluster mask. Trained only on small lattices, the network extrapolates to larger systems, accurately recovers the spanning cluster, and reproduces finite‑size scaling near the critical point, demonstrating that the latent representation captures critical fluctuations and scale‑dependent flows consistent with renormalization‑group theory.

By Anaclara Alvez, Luca Camagna, Sergio Chibbaro, Cyril Furtlehner, Fran\c{c}ois Landes, Gianluca Manzan, Lorenzo Mensi
More like this →