Behavioral Convergence Without Representational Convergence: Persistent Training-History Dependence in Neural Networks
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
The Flow has not summarised this story yet — read it at arXiv Machine Learning.
The study investigates whether a neural network’s training history can better predict its future learning than its current state. Using small multilayer perceptrons trained under 42 different history regimes, the authors measured future learning at four checkpoints via a short probe that replicated the network for 100 updates on a new task. Results show that a low‑dimensional history state does not outperform a calibrated model of the current state for short‑term prediction, but in a companion screen of 1,560 synthetic regression runs, history models forecast final error better than current validation error after 12 epochs and remain comparable until 48 epochs.
arXiv:2608.20965v1 Announce Type: new Abstract: We define an atomic generation fact f=(u,tau,omega,z;rho), recording the origin, realized transformation, concrete occurrence, generated result and rel...
arXiv:2609.36081v1 Announce Type: new Abstract: Representations continually change as a network learns new tasks. We ask whether early representational changes naturally form a geometric structure th...
arXiv:2610.16268v2 Announce Type: replace Abstract: Test-time-training (TTT) layers update fast-weight memory while processing an input sequence. A gradient-based update, however, does not by itself...
arXiv:2507. 01414v2 Announce Type: replace Abstract: We introduce a new family of toy problems that combine features of linear-regression-style continuous in-context learning (ICL) with discrete associative recall.
arXiv:2608. 11690v1 Announce Type: new Abstract: Continual learning must absorb new tasks without erasing old ones, and replay---mixing a small buffer of past examples into current training---is among the most effective remedies for catastrophic forgetting.