The study investigates whether a neural network’s training history can better predict its future learning than its current state. Using small multilayer perceptrons trained under 42 different history regimes, the authors measured future learning at four checkpoints via a short probe that replicated the network for 100 updates on a new task. Results show that a low‑dimensional history state does not outperform a calibrated model of the current state for short‑term prediction, but in a companion screen of 1,560 synthetic regression runs, history models forecast final error better than current validation error after 12 epochs and remain comparable until 48 epochs.
arXiv:2609.37836v1 Announce Type: new Abstract: Neural networks trained toward the same final objective can reach similar predictive performance while retaining internal representations shaped by ear...
arXiv:2610.16268v2 Announce Type: replace Abstract: Test-time-training (TTT) layers update fast-weight memory while processing an input sequence. A gradient-based update, however, does not by itself...
arXiv:2606. 31495v1 Announce Type: new Abstract: We study a single idea across two settings: that a prediction-error signal, computed by a small predictor over the latent space of a frozen encoder, can serve both as a gate on plasticity and as a substrate for metacognition.
The study audits the predictive value of historical summaries in a rule-based university trajectory simulator. Seventeen ridge specifications and persistence models compare current plans, past-state and past-plan summaries, and calendar-only history-length indicators across training, validation, and two family-held-out evaluations, reconstructing 16,750 routes and 804,000 monthly records. Results show modest RMSE improvements with six-month history versus current-state prediction, but negligible differences from a length-only control, with uncertain generalization to all-history summaries and notable violations in fatigue predictions.
The paper investigates what information must be preserved in replay buffers for class‑incremental learning. By treating cached predictions as temporally heterogeneous supervision, the authors separate classes known at storage time from those learned later, and evaluate the impact of deleting logit matching. Experiments on CIFAR‑100 with DER++ show that a simple task‑level offset can largely correct the cost of removing later‑class matching, while the cost of disrupting class correspondence remains.
arXiv:2609.36375v1 Announce Type: new Abstract: Continual learning is usually studied through mechanisms that preserve old knowledge. We develop Successional Learning Theory (SLT), a mesoscopic accou...
The paper argues that the policy governing how tensors are rounded and reused during the backward pass—termed the backward‑state policy—is an integral part of the learning algorithm, not merely a memory or precision detail. Experiments with 390 M‑parameter models show that whether the backward pass reuses a forward’s rounded output or generates a new rounding can decisively affect training success, even when copy accuracy is high. The authors propose a method to determine, for each operator, which value should be read or substituted to preserve the correct gradient, and validate these predictions on PyTorch and Transformer Engine.
Prospective Hindsight (PH) is a self‑calibrating training principle for reinforcement learning that augments any retrospective base method with a signal derived from the gap between an agent’s prospective prediction and its retrospective evaluation. By weighting gradients with a stop‑gradient surprise based on this prediction‑reality gap, PH highlights samples where the agent’s self‑model is most inaccurate and encourages the policy and predictor to co‑evolve, progressively focusing on remaining blind spots. Experiments on single‑turn verifiable tasks and a multi‑turn personal‑agent task show that PH improves both task performance and calibration across model scales, addressing overconfident failures in single‑turn settings and underconfident successes in multi‑turn settings.
arXiv:2610.19768v1 Announce Type: new Abstract: Looped transformers compute by applying one block repeatedly, so they can run for more loops than they were trained with. Recurrent reasoners are known...
arXiv:2607. 25063v1 Announce Type: new Abstract: Developers judge a model checkpoint by how it behaves.
arXiv:2608.24593v1 Announce Type: new Abstract: Adaptive optimizers retain gradient history in moment variables, allowing a local change in loss weighting to alter later updates. We examine whether t...
arXiv:2610.22381v1 Announce Type: new Abstract: When a fact becomes harder to elicit from a language model, has the model lost it, or has it merely become less accessible? Parameter-updating test-tim...