The paper shows that a memory state that reproduces every current prediction can still fail to learn from subsequent observations. It characterizes when a fixed‑dimensional continuous recurrent state exists for exact cumulative learning and attention, proving that for smooth convex residual losses the state exists only when the loss is polynomial. The work further quantifies the dimensionality required for various loss functions, demonstrates unbounded requirements for nonpolynomial losses, and analyzes specific cases such as ridge regression and kernel attention.
arXiv:2607. 19390v1 Announce Type: new Abstract: A recent report finds that orthogonalizing the mLSTM memory matrix at read time (five Newton-Schulz iterations, trained through) substantially improves noisy associative recall.
The paper investigates what a service must retain to faithfully reproduce a training pipeline after data deletion requests, focusing on a rule where earlier similar records block later ones. It shows that a limit on future deletions precisely determines which records can still be needed for training, enabling an exact repair of training statistics after repeated deletions. The authors also construct finite‑precision features for ridge regression and demonstrate that the required private information can be minimized across a sequence of deletion requests, validating their approach on 1,001 AskUbuntu records.
arXiv:2602.06797v3 Announce Type: replace-cross Abstract: We study optimal learning rate (LR) schedules under the functional scaling law (FSL) framework (Li et al., 2025), which decomposes training d...
arXiv:2610. 08816v1 Announce Type: new Abstract: Long-range temporal dependence poses a resource question for sequence models: for a specified predictive-memory law, how much state, context, or dynamical criticality is required in order to forecast accurately?
Fast Weight Attention for Continual Learning introduces recurrent fast‑weight memories and selective state‑space models that compress expanding context into a fixed‑size recurrent state, enabling an online learning rule for state transitions. The paper derives normalized first‑order updates for squared‑error regression and negative inner‑product objectives, presenting several variants (Falcon‑1, Falcon‑2, Falcon‑3 and their inner‑product counterparts) with recurrent, masked‑parallel, and chunk‑parallel implementations. These methods demonstrate competitive performance in language modeling and improved length extrapolation on variable‑digit addition tasks.
arXiv:2606. 30067v1 Announce Type: cross Abstract: We introduce Neural Subspace Reallocation (NSR), which reframes continual learning as memory management over parameter subspaces.
We introduce Neural Subspace Reallocation (NSR), which reframes continual learning as memory management over parameter subspaces. Instead of treating Low-Rank Adaptation (LoRA) modules as disposable per-task adapters, NSR manages them as compressible, retrievable memory units on a frozen backbone through a recurring cycle: (1) compress learned LoRAs via SVD, (2) reserve them in a TaskKnowledgeBank, (3) recall related past LoRAs by embedding similarity to warm-start new or returning tasks, and (4) reallocate the active subspace accordingly, with distillation protecting prior tasks.
arXiv:2610.06894v1 Announce Type: cross Abstract: A central question in the prediction of stochastic processes is the extent to which past information can improve the probability of correctly predict...
arXiv:2606. 21253v2 Announce Type: replace Abstract: Continual learning that is gradient-free, local, online, and append-only is attractive for edge and streaming deployment, but its value is usually argued informally.
arXiv:2607. 05609v1 Announce Type: cross Abstract: The Continual Learning (CL) literature has long been driven by the goal of mitigating catastrophic forgetting.
The paper investigates continual linear regression in high-dimensional settings, deriving a closed-form expression for the generalization loss when using gradient descent with a fixed number of steps per task. It shows that the optimal fixed step size decays roughly as “ln T / T” for a single teacher and i.i.d. teachers, matching the performance of optimally tuned L2 regularization. The study also extends to generalized ridge regression with diagonal regularizers, identifies conditions for isotropic optimality, and reveals a regime where the step size stays finite as the number of tasks grows, highlighting a tradeoff between task variability and implicit regularization.
arXiv:2601. 19791v4 Announce Type: replace Abstract: We study grokking, the onset of generalization long after overfitting, in a classical ridge regression setting.