The paper investigates gradient descent dynamics in a minimal two‑neuron ReLU teacher‑student model with isotropic Gaussian inputs. It identifies two key directions—an easy bisector direction carrying the main signal and a hard splitting direction governing specialization—and shows that every critical point with non‑zero student weights is either a global minimum or a saddle. The study tracks the gradient‑flow trajectory, revealing that student neurons initially converge to a bisector saddle before escaping and specializing to match the teacher neurons, thereby explaining the multi‑phase symmetry‑breaking behavior in this simple setting.
arXiv:2510.25060v2 Announce Type: replace-cross Abstract: In this work, we study the nonlinear dynamics of a shallow neural network trained with mean-squared loss and leaky ReLU activation. Under Gau...
arXiv:2608. 06766v1 Announce Type: cross Abstract: Training changes a network's predictions while allocating task-relevant structure across its internal units.
arXiv:2609.36240v1 Announce Type: new Abstract: Does representation learning stop when the training loss stops improving? We study this question for matrix Muon, whose polar-normalised updates have a...
arXiv:2606. 04476v1 Announce Type: new Abstract: In this paper, we study the gradient descent dynamics for jointly training both layers of a one-hidden-layer ReLU network to fit a linear target function.
The paper investigates the loss landscape of shallow, bias‑free ReLU networks in a teacher–student setting, focusing on Gaussian population loss. It shows that adding a learned linear skip connection eliminates all spurious local minima with non‑negative student output weights when the student network is at least as wide as the teacher, whereas without the skip such minima persist even under overparameterization. Additionally, the study proves that the student network always learns the teacher’s feature subspace and that, in two dimensions, the effective width of an overparameterized student is bounded by twice the teacher’s width.
arXiv:2607. 07845v1 Announce Type: new Abstract: The Hessian of the training loss governs the local geometry of the loss landscape, yet despite existing explanations for its largest eigenvalues, the origin of the vast multitude of vanishingly small eigenvalues remains elusive.
The paper investigates how gradient descent behaves near codimension‑one bifurcations in recurrent neural networks by analyzing the global empirical Neural Tangent Kernel (GeNTK). Under local center‑manifold conditions, the parameter‑to‑state Jacobian is approximated by a low‑rank normal‑form operator, causing the GeNTK and Fisher information matrix to become strongly amplified and anisotropic, concentrating on a rank‑one or rank‑two channel depending on the bifurcation type. Experiments on high‑dimensional RNNs confirm that this low‑rank concentration coincides with abrupt loss changes, subtask interference, and aligns with changes in memory dynamics in a 15‑task LeakyRNN.
The paper investigates how continuous symmetries in neural networks, which give rise to conserved quantities under gradient flow, behave when training with finite‑step gradient descent at the edge of stability (EOS). By applying central flow dynamics, the authors show that curvature corrections cause a finite‑dimensional drift driven by oscillation covariance along critical parameter directions. They analyze this effect in a scalar two‑layer model, two‑layer matrix factorisation, and a two‑layer ReLU network, demonstrating which conservation laws persist or break under EOS conditions.
The paper investigates how two‑layer polynomial‑width neural networks learn orthogonal multi‑index targets under standard initialization. It shows that incremental learning still occurs: the loss decreases sequentially following the Hermite expansion, with lower‑order components learned first. The dynamics also exhibit a competitive reallocation of parameter mass, shifting into the target subspace and concentrating on aligned neurons. The analysis uses a symmetry‑based finite‑width approximation and demonstrates that vanilla gradient descent displays the same qualitative behavior.
arXiv:2606. 04327v1 Announce Type: cross Abstract: We investigate the geometric structure of stationary plateaus that arise in the loss landscape of two-layer neural networks with smooth activation functions.
arXiv:2609.39408v1 Announce Type: cross Abstract: Population loss can remain nearly constant while a neural network learns a substantially more predictive representation. We establish this separation...
arXiv:2607. 03613v1 Announce Type: new Abstract: We study the implicit bias of noisy stochastic gradient descent in training wide two-layer ReLU networks for multivariate regression.