arXiv AI

How Finite Brains Generalize Without Bound: Spatial Memory as a Universal Computer

The paper argues that finite brains generalize to arbitrarily long inputs by using spatial memory that grows with input size while the learner’s observations remain finite. It proposes that this memory is structured as cycles—grid modules, place cells, and hippocampal indices—controlled by a prefrontal system that never directly observes the cycles. Experiments show a neural network trained on strings up to 40 symbols achieves high accuracy on longer sequences, outperforming LSTMs and Transformers on several benchmarks and correctly reversing strings up to 15,015 symbols.

arXiv Machine Learning
Sep 2

How Temporal Correlations Shape Memory in Linear Recurrent Neural Networks

The paper analyzes how temporal correlations in input sequences affect memory formation in linear recurrent neural networks (LRNNs). By solving the learning dynamics for correlated inputs, it shows that correlation introduces a cost to retaining past information, reshaping the learning trajectory and reducing the network’s memory of past inputs. Key findings include a correlation‑dependent threshold for memory retention, the influence of input similarity on memory usefulness, and the emergence of a feedthrough path when zero error is required.

By Arnol Manuel Fokam, Fasseu Sieyondji Akpevwoghene, Edem Fiifi Dawson
arXiv Computation and Language
Nov 9

Representational drift in a code model: behaviour settles, which neurons carry it does not

The study investigates representational drift in the OpenCoder‑1.5B language model by tracking how neurons encode 169 Python constructs during pretraining on two trillion tokens. While the loss for 16 grammar‑fixing constructs quickly reaches a floor, the specific neurons that carry these constructs continue to change throughout training, with no stable carrier set emerging even at the final checkpoint. The authors find that removing the identified carriers incurs a measurable performance penalty, yet the carrier sets remain dynamic, challenging the hypothesis that they settle over time.

By Piotr Wilam
arXiv AI
Oct 16

Learning to Read Out: Unembedding Dynamics in Language Model Pretraining

The paper investigates when a language model actually learns task information versus when its readout (unembedding matrix) learns to express that information. Using a new parameter‑trajectory cross‑coding method, the authors find that hidden states can encode task data early in training, but the readout reorganizes only in a narrow window (0.4–1.1% of Pythia’s pretraining). By varying the readout learning rate, they show that faster readout learning reduces the gap between hidden‑state encoding and output expression without affecting final performance.

By Matteo He, William F. Shen, Alex Iacob, Andrej Jovanovic, Xinchi Qiu, Nicholas D. Lane
arXiv AI
Jun 24

Repeated Shared Access Enables Grokking, but Edit Propagation Depends on an Addressable Memory

arXiv:2606. 20737v2 Announce Type: replace Abstract: We study factual edit propagation in a controlled synthetic knowledge-graph QA setting using a 2x2 grid that crosses loop recurrence with shared-memory access: a dense transformer (Dense), a looped transformer (Loop), a dense backbone with shared memory (Dense+Mem), and a looped backbone with shared memory (loop-memory coupling, LMC).

By Yanan Niu