Invariant learning seeks representations that remain predictive across environments, yet the behavior of its objectives along the regularization path is often opaque. We address this objective-behavior gap by viewing representation learning as multimode magnetization and deriving, from concrete invariant-learning objectives, a Landau-type effective free energy whose low-order coefficients form objective signatures and induce distinct regularization phenotypes.
arXiv:2608. 06597v1 Announce Type: cross Abstract: A scientific theory of deep learning, comprising learning dynamics and statistical properties of learned models, is rapidly gaining attention.
By Bj\"orn Ladewig, Ibrahim Talha Ersoy, Karoline Wiesner
arXiv:2607. 10285v1 Announce Type: new Abstract: We study how unsupervised autoencoders trained on microscopic spin configurations from the Ising model learn macroscopic, theory-relevant variables underlying the data-generating process.
By Max Weinmann, Miriam Klopotek
The paper derives an exact discrete‑time law that captures how learning‑rate schedules and weight decay interact in scale‑invariant neural networks, showing that a single scalar quantity governs the effective step size. It demonstrates that the balance point between contraction and expansion is intrinsically unstable, leading to recurrent dynamics when using constant learning rates with weight decay. The authors extend this analysis to various optimizers and datasets, confirming the law’s precision and showing that performance peaks sharply at the predicted boundary.
By Hasan Amin, Wei-Kai Chang, Rajiv Khanna
arXiv:2608.28059v1 Announce Type: cross
Abstract: In-context learning (ICL) allows a pretrained model to infer a new task from examples supplied in its prompt without updating its parameters. In line...
By Daesik Kim, Sumin Choi, Hyojae Jeon, Jung Hoon Han
The study investigates the delayed transition from memorization to generalization—known as grokking—in two‑hidden‑layer MLPs trained on modular arithmetic. By exploring 384 hyperparameter configurations, the authors derive a power‑law scaling relation for the onset time of generalization, showing that data complexity dominates over model capacity. A clear phase boundary at weight decay around 1.0 separates grokking from non‑grokking regimes, and weight norm trajectories indicate implicit regularization during the transition.
By Anish Kataria
arXiv:2606. 17120v1 Announce Type: new Abstract: Deep neural networks (DNNs) exhibit first order phase transitions under variations of the L2 regularization strength, with each transition marking the onset of a new learnable feature.
By Ibrahim Talha Ersoy, Karoline Wiesner
The paper investigates how normalization makes neural networks scale‑invariant, creating a feedback loop between learning‑rate schedules and weight decay that controls the effective step size of the optimizer. It derives an exact discrete‑time law showing that a single scalar quantity captures all schedule and decay effects, with norm growth providing a self‑quenching counter‑force that defines a sharp boundary between contraction‑ and expansion‑dominated regimes. Through exact analysis of a normalized regression model and experiments on MLPs, CNNs, GPT‑2, and various datasets, the authors demonstrate that constant learning rates with weight decay are intrinsically unstable, leading to recurrent dynamics, and that adaptive optimizers exhibit weaker stabilization under normalization.
"whyItMatters":"The study provides a precise, actionable rule for controlling training dynamics and schedule design in modern deep learning by isolating a single governing quantity for scale‑invariant optimization."
The paper presents a spectral theory explaining the phenomenon of grokking, where an initial fit to training data is followed by a delayed improvement in generalization. It shows that for homogeneous networks trained with squared loss and L₂ weight decay, residuals after memorization influence the neural tangent kernel (NTK) dynamics, leading to a transition from lazy to rich learning. The theory predicts that grokking timescales depend on the product of learning rate and weight decay, and that stronger decay can halt fitting, with empirical validation on modular addition tasks using MLPs and Transformers.
By Lenz Pracher, Pascal de Jong, Oskar Lieshaus, Alan Jeffares, Steffen Rulands
arXiv:2604. 00230v2 Announce Type: replace Abstract: Neural collapse (NC) -- the convergence of penultimate-layer features to a simplex equiangular tight frame -- is well understood at equilibrium, but the dynamics governing its onset remain poorly characterised.
By Anamika Paul Rupa
AYLA is a loss reparameterization framework that applies a sigmoid‑controlled power‑law transformation to the empirical loss, dynamically adjusting gradient magnitudes without changing stationary points or optimal solutions. By reshaping optimization trajectories, AYLA accelerates descent in flat or saddle‑dominated regions and stabilizes late‑stage training, leading to improved feature recovery in two‑layer tanh networks on synthetic Gaussian data. Experiments show enhanced weight alignment, neuron similarity, activation correlation, and richer internal representations, while mitigating rank collapse and promoting a transition from lazy to active feature‑learning regimes.
By Behnam Gheshlaghi, Shahin Atakishiyev
arXiv:2606. 24660v1 Announce Type: cross Abstract: Phase-field models play a central role in the continuum description of phase separation, in which the bulk free-energy density and the interfacial thickness parameter determine pattern formation and microstructural evolution.
By Callum Marsh, Radek Erban, Andreas Munch