The paper investigates continual linear regression in high-dimensional settings, deriving a closed-form expression for the generalization loss when using gradient descent with a fixed number of steps per task. It shows that the optimal fixed step size decays roughly as “ln T / T” for a single teacher and i.i.d. teachers, matching the performance of optimally tuned L2 regularization. The study also extends to generalized ridge regression with diagonal regularizers, identifies conditions for isotropic optimality, and reveals a regime where the step size stays finite as the number of tasks grows, highlighting a tradeoff between task variability and implicit regularization.
By Gilad Karpel, Daniel Soudry
arXiv:2606. 30559v1 Announce Type: new Abstract: We characterize weakly regularized continual classification in homogeneous models as sequential projections onto task margin sets.
By Matan Schliserman, Gon Buzaglo, Itay Evron, Daniel Soudry
The paper investigates the performance cost, termed the sequential price, incurred by sequential task updates in continual learning. In an overparameterized linear‑regression setting with i.i.d. task sampling, the authors prove that distribution‑level forgetting and population loss converge to a stationary limit that splits into the intrinsic loss of joint training and an additional sequential price. They further analyze elastic weight consolidation (EWC), showing how its regularization strength affects the stationary sequential price and demonstrate the theory on the Jester joke‑rating dataset.
By Zonghuan Xu, Xingjun Ma
arXiv:2610.32708v1 Announce Type: new
Abstract: Continual learning (CL) aims to learn from sequential tasks while mitigating catastrophic forgetting when historical data are no longer accessible. We...
By Jingyi Guo, Runze Li, Lingzhou Xue
Continual training of deep neural networks under non-stationarity often leads to a progressive loss of plasticity, eventually limiting further learning. We relate plasticity to the empirical Neural Tangent Kernel, and identify dynamical isometry (the condition that layer-wise Jacobian singular values remain close to one) as a key mechanism for preserving plasticity in continual learning.
arXiv:2606. 09762v1 Announce Type: cross Abstract: Continual training of deep neural networks under non-stationarity often leads to a progressive loss of plasticity, eventually limiting further learning.
By Andries Rosseau, Robert M\"uller, Ann Now\'e
arXiv:2606. 26629v1 Announce Type: new Abstract: Weight-space regularization methods such as Elastic Weight Consolidation (EWC) are the standard approach to catastrophic forgetting in continual learning.
By Evan Ning, Wei Xue, Dong Lou, Yike Guo
The paper introduces an algorithmic stability framework to analyze catastrophic forgetting in continual learning using SGD, without assuming a specific model architecture. It decomposes forgetting into intra‑task and cross‑task generalization gaps plus empirical forgetting, and derives bounds for different learning‑rate schedules. Experiments on Split CIFAR‑100 and Split MiniImageNet illustrate the theoretical insights.
By Guodong Zheng, Yan Sun, Peng Wang, Enneng Yang, Jiafei Lyu, Xiaochun Cao, Dacheng Tao, Li Shen
arXiv:2510. 12249v2 Announce Type: replace Abstract: In performative learning, the data distribution reacts to the deployed model - for example, because strategic users adapt their features to game it - which creates a more complex dynamic than in classical supervised learning.
By Edwige Cyffers, Alireza Mirrokni, Marco Mondelli
arXiv:2608. 15901v1 Announce Type: cross Abstract: Continual learning regularizers like EWC fight forgetting by penalizing changes from previous-task parameters with per-parameter importance, typically diagonal Fisher values.
By Brian B. Moser, Ahmed Anwar, Tobias Christian Nauen, Shishir Muralidhara, Federico Raue, Ren\'e Schuster, Stanislav Frolov, Andreas Dengel
arXiv:2609.17026v1 Announce Type: new
Abstract: Continual learning must balance the learning of new knowledge with the retention of previously learned knowledge to incrementally learn tasks from a da...
By Yunxiang Fu, Meng Lou, Zicheng Liao, Yizhou Yu
arXiv:2608. 01475v1 Announce Type: new Abstract: Neural networks that can grow or both grow and shrink during learning, referred to as growing neural networks and elastic neural networks, respectively, have recently been explored in offline continual learning with a particular focus on catastrophic forgetting.
By Jeong Min Kong, Richard S. Sutton