arXiv Machine Learning

How Temporal Correlations Shape Memory in Linear Recurrent Neural Networks

The paper analyzes how temporal correlations in input sequences affect memory formation in linear recurrent neural networks (LRNNs). By solving the learning dynamics for correlated inputs, it shows that correlation introduces a cost to retaining past information, reshaping the learning trajectory and reducing the network’s memory of past inputs. Key findings include a correlation‑dependent threshold for memory retention, the influence of input similarity on memory usefulness, and the emergence of a feedthrough path when zero error is required.

arXiv AI
Nov 6

How Finite Brains Generalize Without Bound: Spatial Memory as a Universal Computer

The paper argues that finite brains generalize to arbitrarily long inputs by using spatial memory that grows with input size while the learner’s observations remain finite. It proposes that this memory is structured as cycles—grid modules, place cells, and hippocampal indices—controlled by a prefrontal system that never directly observes the cycles. Experiments show a neural network trained on strings up to 40 symbols achieves high accuracy on longer sequences, outperforming LSTMs and Transformers on several benchmarks and correctly reversing strings up to 15,015 symbols.

By Yuhang He, Si Wu
arXiv AI
Oct 16

Learning to Read Out: Unembedding Dynamics in Language Model Pretraining

The paper investigates when a language model actually learns task information versus when its readout (unembedding matrix) learns to express that information. Using a new parameter‑trajectory cross‑coding method, the authors find that hidden states can encode task data early in training, but the readout reorganizes only in a narrow window (0.4–1.1% of Pythia’s pretraining). By varying the readout learning rate, they show that faster readout learning reduces the gap between hidden‑state encoding and output expression without affecting final performance.

By Matteo He, William F. Shen, Alex Iacob, Andrej Jovanovic, Xinchi Qiu, Nicholas D. Lane
arXiv AI
Sep 1

On the Plasticity Collapse in Continual Machine Unlearning

The paper investigates continual machine unlearning, where models must forget data over time. It identifies a fundamental issue called plasticity collapse, where successive unlearning requests cause geometric constraints that saturate parameter space, leading to two failure modes: forward failure (reduced forgetting quality) and backward failure (re‑memorization). Experiments across architectures and datasets confirm that plasticity collapse is a pervasive problem in continual unlearning.

By Yingdan Shi, Xiang Xu, Kaize Ding, Alfred O. Hero, Ren Wang
arXiv Machine Learning
Oct 8

When does a network's training history predict its future learning better than its current state? Evidence from a response probe and a forecasting screen

The study investigates whether a neural network’s training history can better predict its future learning than its current state. Using small multilayer perceptrons trained under 42 different history regimes, the authors measured future learning at four checkpoints via a short probe that replicated the network for 100 updates on a new task. Results show that a low‑dimensional history state does not outperform a calibrated model of the current state for short‑term prediction, but in a companion screen of 1,560 synthetic regression runs, history models forecast final error better than current validation error after 12 epochs and remain comparable until 48 epochs.

By Martin Hofmann, Patrick M\"ader
arXiv AI
Oct 27

Streaming Recurrent Memory

arXiv:2610.34833v1 Announce Type: cross Abstract: Natural intelligence learns from a continuous stream of sensorimotor experience: data can arrive at high bandwidth, and behavior adapts to feedback a...

By Mohamed Elsayed, Minh Pham, A. Rupam Mahmood
arXiv Machine Learning
Sep 18

Learning-Induced Dynamical Transition in Recurrent Neural Networks

The paper presents a non-equilibrium dynamical mean-field theory (DMFT) that explains how learning reshapes the dynamics of recurrent neural networks, turning initially chaotic activity into stable, task-dependent behavior. It shows that a slow, feedback-driven learning process gradually increases effective feedback strength, driving the network through a bifurcation that marks the transition from chaotic to stable dynamics. By deriving the two-time correlation function, the authors identify a critical feedback strength and a learning-rate-dependent critical time that separate these regimes, and they demonstrate that the theory accurately predicts the network’s output evolution during training, matching numerical simulations.

By Varun Vaidya