arXiv Machine Learning

Center-Manifold Reduction of Learning at Bifurcations: Interference and Rich Learning in Recurrent Neural Networks

The paper investigates how gradient descent behaves near codimension‑one bifurcations in recurrent neural networks by analyzing the global empirical Neural Tangent Kernel (GeNTK). Under local center‑manifold conditions, the parameter‑to‑state Jacobian is approximated by a low‑rank normal‑form operator, causing the GeNTK and Fisher information matrix to become strongly amplified and anisotropic, concentrating on a rank‑one or rank‑two channel depending on the bifurcation type. Experiments on high‑dimensional RNNs confirm that this low‑rank concentration coincides with abrupt loss changes, subtask interference, and aligns with changes in memory dynamics in a 15‑task LeakyRNN.

arXiv AI

Aug 25

Divisive Normalization Shapes Low-Rank Slow Manifolds for Continuous Working Memory

The paper introduces the Recurrent Divisive Normalization Network (RDNN), a minimal model that incorporates divisive normalization—a common neural computation—to stabilize continuous working memory representations. Dynamical systems analysis shows that this biophysical constraint enables the network to converge to robust, high‑fidelity slow manifolds, while gradient dynamics during Backpropagation Through Time reveal an activity‑dependent local scaling that compresses the network’s effective rank into a low‑dimensional subspace. Ablation studies confirm that divisive normalization, rather than subtractive inhibition, is essential for preventing manifold shattering under time‑varying inputs.

By Zhaotian Gu, Jie Su, Weiwei Wang, Chang Liu, Tianyi Qian, Dahui Wang

fine-tuning

arXiv Machine Learning

Jun 30

Scalar Representations of Neural Network Training Dynamics

arXiv:2606. 30384v1 Announce Type: new Abstract: Training in artificial neural networks can be viewed as a trajectory evolving through a high-dimensional loss landscape.

By Pedro Jim\'enez-Gonz\'alez, Miguel C. Soriano, Lucas Lacasa

rag

arXiv Machine Learning

Jun 16

A Bifurcation Theory Framework for Gradient Descent on the Edge of Stability

arXiv:2606. 15551v1 Announce Type: new Abstract: The Edge of Stability (EoS) phenomenon, where gradient descent operates with sharpness exceeding the classical convergence threshold yet the loss decreases over long timescales, is ubiquitous in modern deep learning but remains poorly understood in realistic settings.

By Eric Gan

diffusion

Hugging Face Trending Papers

Jun 29

Scalar Representations of Neural Network Training Dynamics

Training in artificial neural networks can be viewed as a trajectory evolving through a high-dimensional loss landscape. However, the large number of trainable parameters makes the direct analysis of these dynamics challenging.

rag

arXiv Machine Learning

Aug 3

Embedding of Low-Dimensional Sensory Dynamics in Recurrent Networks: Implications for the Geometry of Neural Representation

arXiv:2601. 19019v3 Announce Type: replace-cross Abstract: Neural population activity in sensory cortex is organized on low-dimensional manifolds, but why such manifolds arise and what determines their geometry remain unclear.

By Vikas N. O'Reilly-Shah, Alessandro Maria Selvitella

rag

arXiv AI

Jun 10

Machine Learning Methods for Studying Latent Neural Activity Dynamics

arXiv:2606. 10530v1 Announce Type: cross Abstract: Recent developments in brain recording are driving a demand for machine learning tools capable of decoding the latent structure of large populations of neurons.

By Shufeng Kong, Fumei Deng, Xinyi Dong, Caihua Liu, Weiwei Chen, Yingheng Wang, Daniel Cao, Azahara Oliva, Antonio Fernandez-Ruiz, Carla Gomes

llms diffusion benchmarks

arXiv Machine Learning

Sep 15

Random matrix theory of sparse neuronal networks with heterogeneous timescales

arXiv:2512.12767v2 Announce Type: replace-cross Abstract: Training recurrent neuronal networks consisting of excitatory (E) and inhibitory (I) units with additive noise for working memory computation...

By Thiparat Chotibut, Oleg Evnin, Weerawit Horinouchi

efficiency

arXiv Machine Learning

Jul 3

Dendritic In-Context Learning in a Single-Layer Spiking Neural Network

arXiv:2607. 02283v1 Announce Type: cross Abstract: In-context learning (ICL) operates via implicit gradient descent embedded in the forward pass of modern AI architectures -- Transformers, Mamba, state-space models, and MLPs.

By Juwei Shen, Yujie Wu, Changwen Chen

llms benchmarks

arXiv AI

Jul 16

Transforming Rank: How Architecture Navigates the Spectral Pathologies of Depth

arXiv:2607. 14018v1 Announce Type: cross Abstract: We investigate how each component of the Transformer feedforward block architecture design determines how much rank survives across depth at initialization.

By Katie Everett

llms

Hugging Face Trending Papers

Jul 15

Transforming Rank: How Architecture Navigates the Spectral Pathologies of Depth

We investigate how each component of the Transformer feedforward block architecture design determines how much rank survives across depth at initialization. We reinterpret skip connections and normalization, long understood as controlling magnitude, as mechanisms for preserving gradient rank across depth, since the very matrix multiplications and nonlinear activations that make the network expressive also reduce the rank.

llms

Hugging Face Trending Papers

Sep 21

Ananke: Contractive Torus Attractor Networks

We introduce Ananke, a representation-learning framework that scaffolds latent representations onto a structured product-torus prior, and its flagship visual backbone realization, Contractive Torus At...

diffusion benchmarks

arXiv AI

Jun 10

Between Amnesia and Chaos: A Memory Stability Expressivity Trilemma for Trainable Dissipative Oscillator Networks

arXiv:2606. 09929v1 Announce Type: cross Abstract: Physical reservoir computing harnesses nonlinear mechanical dynamics but, by convention, freezes the substrate and trains only a linear readout, presuming the substrate is not usefully trainable.

By Caleb Munigety

Related stories