The paper investigates how learning algorithms behave when gradient signals are noisy, sparse, and intermittently available, as often occurs in biological neural networks. It demonstrates that even if the average learning dynamics converge to the correct optimum, the variance of learning can become asymptotically infinite, leading to instability. The authors analyze several analytically tractable settings—including single-layer and multi-layer linear networks—to derive explicit conditions under which learning remains stable, highlighting that unbiased credit assignment alone is insufficient without sufficiently frequent and informative feedback.
arXiv:2607. 03613v1 Announce Type: new Abstract: We study the implicit bias of noisy stochastic gradient descent in training wide two-layer ReLU networks for multivariate regression.
arXiv:2609.06430v1 Announce Type: new Abstract: We study the identity straight-through estimator (STE) for training a two-layer binary-activation network with hinge loss from the perspective of Stati...
The paper analyzes Gradient Descent (GD) on a minimal binary classification problem with a two‑neuron ReLU network trained on two instances. It proves that GD converges to an optimal robustness margin—maximizing the distance between the decision boundary and training points—but does so at a very slow rate, specifically θ(1/ln(t)). Empirical simulations confirm this slow convergence across various initializations, and the authors provide a rigorous analysis of GD trajectories across different activation patterns to establish the first explicit lower bound on robustness convergence in a non‑linear model.
arXiv:2606. 04476v1 Announce Type: new Abstract: In this paper, we study the gradient descent dynamics for jointly training both layers of a one-hidden-layer ReLU network to fit a linear target function.
arXiv:2610. 23007v1 Announce Type: new Abstract: Gradient descent (GD) with large learning rates can successfully optimize neural networks even though classical smoothness-based learning rate conditions are violated.
arXiv:2606. 30930v1 Announce Type: cross Abstract: Modern deep learning has been shown to operate at the edge of stability, routinely using learning rates far larger than those justified by classical optimization theory.
arXiv:2507.10383v5 Announce Type: replace-cross Abstract: Recurrent neural networks are canonical models of biological memory. In these models, memories are represented by distributed patterns of neu...
arXiv:2507. 05164v2 Announce Type: replace-cross Abstract: In this chapter, we utilize dynamical systems to analyze several aspects of machine learning algorithms.
arXiv:2506. 08764v3 Announce Type: replace Abstract: Deep neural networks are known to suffer from exploding or vanishing gradients as depth increases, a phenomenon closely tied to the spectral behavior of the input-output Jacobian.
arXiv:2401. 04013v2 Announce Type: replace Abstract: Deep learning models, such as wide neural networks, can be conceptualized as nonlinear dynamical physical systems characterized by a multitude of interacting degrees of freedom.
arXiv:2501. 07400v2 Announce Type: replace-cross Abstract: We derive explicit equations governing the cumulative biases and weights in Deep Learning with ReLU activation function, based on gradient descent for the Euclidean loss in the input layer, and under the assumption that the weights are, in a precise sense, adapted to the coordinate system distinguished by the activations.
The paper presents convergence guarantees for Stochastic Gradient Descent (SGD) applied to the multiclass cross-entropy loss in both linear classifiers and two-layer neural networks. It shows that SGD dynamics can oscillate between an edge-of-stability regime dominated by curvature-driven oscillations and a stable regime where the expected loss decreases at a controlled rate. Despite these oscillations, the authors prove that SGD stabilizes the dynamics, ensuring that iterates return to stability within a fixed number of iterations and enabling convergence in the best-iterate sense even with large learning rates.