The paper investigates how continuous symmetries in neural networks, which give rise to conserved quantities under gradient flow, behave when training with finite‑step gradient descent at the edge of stability (EOS). By applying central flow dynamics, the authors show that curvature corrections cause a finite‑dimensional drift driven by oscillation covariance along critical parameter directions. They analyze this effect in a scalar two‑layer model, two‑layer matrix factorisation, and a two‑layer ReLU network, demonstrating which conservation laws persist or break under EOS conditions.
arXiv:2606. 18080v1 Announce Type: new Abstract: Gradient descent in deep learning may operate at the edge of stability (EoS), a regime in which the largest eigenvalue of the loss Hessian hovers near the stability threshold $2/\eta$, where $\eta$ is the learning rate.
arXiv:2606. 10913v1 Announce Type: new Abstract: We explore whether intrinsic symmetries of the training data lead to conserved quantities during gradient-flow training of neural networks.
arXiv:2609.01034v1 Announce Type: new Abstract: The central flow of Cohen et al. (2025) is an empirically accurate continuous-time model of gradient descent at the edge of stability in deep learning,...
The paper introduces the edge coupling, a functional whose critical points correspond to the fixed points and period‑two orbits of gradient descent. It shows that both central flow and rod flow—previous continuous‑time models for the edge of stability—can be derived from this edge coupling, with rod flow coarse‑graining only the fastest time scale and central flow coarse‑graining both fast and intermediate scales. The authors extend these derivations to momentum methods such as heavy ball and Adam, and experimentally confirm that the resulting flows track discrete iterates across all training phases, including the transition to the edge of stability.
arXiv:2606. 05326v1 Announce Type: cross Abstract: We study the dynamics of gradient descent in the Edge of Stability regime, where the learning rate is large enough to induce persistent oscillations in the loss and the sharpness.
arXiv:2610. 09356v1 Announce Type: new Abstract: We prove global exponential (linear) convergence with an explicit rate in the rich scaling for wide two-layer linear networks trained with smooth Polyak-Lojasiewicz predictor losses.
The paper investigates gradient descent dynamics in the Edge of Stability regime, where a large learning rate causes persistent oscillations linked to improved generalization. It introduces a tractable continuous‑time mean–fluctuation model that couples the window‑averaged trajectory with its fluctuation covariance, derives this model rigorously from a sharp‑valley framework, and analyzes its stationary states and linear stability. The authors also extend the model to wide two‑layer networks, deriving a Wasserstein‑2 gradient flow for weights and fluctuations, proving well‑posedness, a mean‑field limit, and conditional convergence results, with numerical experiments illustrating the predictions and finite‑time limitations.
The paper investigates how finite‑stepsize stochastic gradient descent (SGD) breaks scale symmetry in undercomplete linear autoencoders. It shows that SGD induces a directed scale drift that favors large decoder weights on the PCA solution manifold, a process that is slow and analytically tractable but eventually halted by a stability boundary. The resulting solutions exhibit sharper loss Hessians compared to a balanced baseline, illustrating how loss geometry can transform residual gradient noise into directed motion along equivalent solution manifolds.
arXiv:2606. 17816v1 Announce Type: cross Abstract: Understanding gradient descent dynamics is key to explaining the success of over-parameterized models, where implicit bias manifests through conservation laws in gradient flow.
arXiv:2607. 07845v1 Announce Type: new Abstract: The Hessian of the training loss governs the local geometry of the loss landscape, yet despite existing explanations for its largest eigenvalues, the origin of the vast multitude of vanishingly small eigenvalues remains elusive.
The paper introduces a diagnostic tool that measures how neural emulators of partial differential equations capture physical symmetries by evaluating the overlap of loss gradients along symmetry-related states. This metric probes the local geometry of the learned loss landscape and goes beyond traditional equivariance tests by directly assessing learning dynamics. Applied to autoregressive fluid flow emulators, the study shows that orbit-wise gradient coherence enables generalization over symmetry transformations and reveals when training selects a symmetry-compatible basin.
The paper investigates gradient descent dynamics in a minimal two‑neuron ReLU teacher‑student model with isotropic Gaussian inputs. It identifies two key directions—an easy bisector direction carrying the main signal and a hard splitting direction governing specialization—and shows that every critical point with non‑zero student weights is either a global minimum or a saddle. The study tracks the gradient‑flow trajectory, revealing that student neurons initially converge to a bisector saddle before escaping and specializing to match the teacher neurons, thereby explaining the multi‑phase symmetry‑breaking behavior in this simple setting.