The paper presents a theoretical study of how intrinsic noise affects learning dynamics in noisy linear recurrent neural networks. It demonstrates that noise can break degeneracies in the solution space, influence the alignment of read-in, readout, and recurrent weights, and even induce a phase transition where a non-degenerate solution becomes suboptimal under high noise and parameter constraints. These findings suggest that noise is not merely a nuisance but can fundamentally shape representation learning in RNNs.
By John J. Vastola, Kanaka Rajan
The paper investigates how gradient descent behaves near codimension‑one bifurcations in recurrent neural networks by analyzing the global empirical Neural Tangent Kernel (GeNTK). Under local center‑manifold conditions, the parameter‑to‑state Jacobian is approximated by a low‑rank normal‑form operator, causing the GeNTK and Fisher information matrix to become strongly amplified and anisotropic, concentrating on a rank‑one or rank‑two channel depending on the bifurcation type. Experiments on high‑dimensional RNNs confirm that this low‑rank concentration coincides with abrupt loss changes, subtask interference, and aligns with changes in memory dynamics in a 15‑task LeakyRNN.
By James Hazelden, Eric Shea-Brown
arXiv:2511. 13899v2 Announce Type: replace-cross Abstract: Low-rank recurrent neural networks (lrRNNs) are a class of models that uncover low-dimensional latent dynamics underlying neural population activity.
By Chengrui Li, Yunmiao Wang, Yule Wang, Weihan Li, Dieter Jaeger, Anqi Wu
arXiv:2606. 30384v1 Announce Type: new Abstract: Training in artificial neural networks can be viewed as a trajectory evolving through a high-dimensional loss landscape.
By Pedro Jim\'enez-Gonz\'alez, Miguel C. Soriano, Lucas Lacasa
arXiv:2606. 10530v1 Announce Type: cross Abstract: Recent developments in brain recording are driving a demand for machine learning tools capable of decoding the latent structure of large populations of neurons.
By Shufeng Kong, Fumei Deng, Xinyi Dong, Caihua Liu, Weiwei Chen, Yingheng Wang, Daniel Cao, Azahara Oliva, Antonio Fernandez-Ruiz, Carla Gomes
Training in artificial neural networks can be viewed as a trajectory evolving through a high-dimensional loss landscape. However, the large number of trainable parameters makes the direct analysis of these dynamics challenging.
arXiv:2601. 04539v2 Announce Type: replace-cross Abstract: In recurrent neural networks (RNNs) used to model biological neural networks, noise is typically introduced during training to emulate biological variability and regularize learning.
By Noah Eckstein, Manoj Srinivasan
arXiv:2610.23779v1 Announce Type: cross
Abstract: Understanding how stochastic gradient descent learns internal representations in wide neural networks trained on rich target functions is a central p...
By Shu Zhou, Jean Barbier, Sebastian Goldt
arXiv:2610.31783v1 Announce Type: new
Abstract: We study the early training dynamics of nonlinear residual networks with blocks of zero hidden layers as a model of more general residual networks. We...
By Vladimir Narovlansky, Boris Hanin
arXiv:2607. 08843v1 Announce Type: new Abstract: In artificial and biological neural networks, concepts are often encoded as consistent linear directions in representation space.
By William W. Yang, Andrew M. Saxe, Peter E. Latham
The paper investigates the relationship between representation and function in neural networks, using analytical two‑layer linear models and simulations of nonlinear networks. It demonstrates that functional similarity and representational similarity can be dissociated: networks may share representations without sharing functions, and vice versa. The study also finds that robustness to input noise or generalization error does not constrain representations, whereas robustness to parameter noise forces networks to adopt task‑specific representations, indicating that representational alignment reflects computational advantages beyond mere functional alignment.
By Lukas Braun, Erin Grant, Andrew M. Saxe
The paper proposes a second‑order recurrent neural network that replaces traditional neuron‑to‑neuron communication with a spatially evolving field governed by discretized partial differential equations. This design creates a structured infinite‑order RNN where each state depends on its entire history, providing an unbounded receptive field while keeping the number of parameters fixed. The authors derive conditions for marginal stability that prevent vanishing or exploding gradients, and demonstrate that the architecture outperforms other recurrent models on long‑horizon benchmarks with fewer parameters.
By Aleix Salvador-Pomarol, Arthur N. Montanari, Earl K. Miller, Adilson E. Motter, Jorge Cort\'es