arXiv:2611.06759v1 Announce Type: new Abstract: We study logistic regression for continual learning with linear predictors in a high-dimensional statistical setting. Existing theoretical work has lar...
arXiv:2601. 13844v3 Announce Type: replace Abstract: We study generalization in an overparameterized continual linear regression setting, where a model is trained with L2 (isotropic) regularization across a sequence of tasks.
The paper investigates continual linear regression in high-dimensional settings, deriving a closed-form expression for the generalization loss when using gradient descent with a fixed number of steps per task. It shows that the optimal fixed step size decays roughly as “ln T / T” for a single teacher and i.i.d. teachers, matching the performance of optimally tuned L2 regularization. The study also extends to generalized ridge regression with diagonal regularizers, identifies conditions for isotropic optimality, and reveals a regime where the step size stays finite as the number of tasks grows, highlighting a tradeoff between task variability and implicit regularization.
arXiv:2610.32708v1 Announce Type: new Abstract: Continual learning (CL) aims to learn from sequential tasks while mitigating catastrophic forgetting when historical data are no longer accessible. We...
arXiv:2606. 26629v1 Announce Type: new Abstract: Weight-space regularization methods such as Elastic Weight Consolidation (EWC) are the standard approach to catastrophic forgetting in continual learning.
arXiv:2606. 30559v1 Announce Type: new Abstract: We characterize weakly regularized continual classification in homogeneous models as sequential projections onto task margin sets.
arXiv:2411. 16073v4 Announce Type: replace-cross Abstract: Inspired by the Well-initialized Lottery Ticket Hypothesis (WLTH), we introduce Soft-TransFormers (Soft-TF), a continual learning framework that adapts a frozen pre-trained Transformer through task-specific soft subnetworks: real-valued multiplicative masks over the query, key, value, and output projections of selected self-attention layers.
arXiv:2608. 15901v1 Announce Type: cross Abstract: Continual learning regularizers like EWC fight forgetting by penalizing changes from previous-task parameters with per-parameter importance, typically diagonal Fisher values.
arXiv:2609.17026v1 Announce Type: new Abstract: Continual learning must balance the learning of new knowledge with the retention of previously learned knowledge to incrementally learn tasks from a da...
arXiv:2605.05436v2 Announce Type: replace-cross Abstract: Deep learning systems are known to exhibit implicit regularization (alt. implicit bias), favoring simple solutions instead of merely minimizi...
The paper introduces an algorithmic stability framework to analyze catastrophic forgetting in continual learning using SGD, without assuming a specific model architecture. It decomposes forgetting into intra‑task and cross‑task generalization gaps plus empirical forgetting, and derives bounds for different learning‑rate schedules. Experiments on Split CIFAR‑100 and Split MiniImageNet illustrate the theoretical insights.
The paper argues that catastrophic forgetting and loss of plasticity alone cannot explain why naive sequential training underperforms offline joint training. It introduces data co-observation as a third factor, showing that observing training data together consistently improves performance across supervised and self-supervised settings. The study also reinterprets common continual learning methods, suggesting that memory replay’s success stems from restoring co-observation benefits rather than merely mitigating forgetting.