arXiv AI
Jul 8
To Retain or to Adapt? Generalizing Continual Learning
arXiv:2607. 05609v1 Announce Type: cross Abstract: The Continual Learning (CL) literature has long been driven by the goal of mitigating catastrophic forgetting.
Fine-Grained Analysis of Catastrophic Forgetting in Continual Learning via Algorithmic Stability
The paper introduces an algorithmic stability framework to analyze catastrophic forgetting in continual learning using SGD, without assuming a specific model architecture. It decomposes forgetting into intra‑task and cross‑task generalization gaps plus empirical forgetting, and derives bounds for different learning‑rate schedules. Experiments on Split CIFAR‑100 and Split MiniImageNet illustrate the theoretical insights.
Related stories
arXiv Machine Learning
Aug 13
Drift and Dependence: Layer-wise Information-Theoretic Bounds for Replay-Based Continual Learning
arXiv:2608. 11690v1 Announce Type: new Abstract: Continual learning must absorb new tasks without erasing old ones, and replay---mixing a small buffer of past examples into current training---is among the most effective remedies for catastrophic forgetting.
arXiv Machine Learning
Jun 9
Theoretical Foundations of Continual Learning via Drift-Plus-Penalty
arXiv:2606. 08452v1 Announce Type: new Abstract: In many real-world settings, data streams are nonstationary and arrive sequentially, requiring learning systems to adapt continuously without retraining from scratch.
arXiv AI
Sep 28
Estimating and Orthogonalizing Unknown Pre-training Gradients for Continual Fine-tuning of Large Language Models
The paper introduces EoupCT, a framework that estimates and orthogonalizes unknown pre‑training gradients to mitigate catastrophic forgetting during continual fine‑tuning of large language models. It generates pseudo data most susceptible to forgetting using a learnable soft prompt with Gumbel‑Softmax, then jointly optimizes model parameters and the prompt via a first‑order Pareto optimizer to enforce orthogonality between new task updates and the estimated gradients. Experiments on multiple LLMs show that EoupCT preserves both task‑specific performance and the models’ inherent general‑purpose knowledge.
arXiv Machine Learning
Oct 3
PMF-CL: Pareto-Minimal-Forgetting Continual Learner for Conflicting Tasks
arXiv:2605. 19145v3 Announce Type: replace Abstract: In the literature, many continual learning (CL) algorithms have been proposed to address the issue of catastrophic forgetting in ML models (i.
arXiv Machine Learning
Jun 9
Evaluating the Impact of Task Granularity on Catastrophic Forgetting in Continual Learning
arXiv:2606. 08013v1 Announce Type: new Abstract: Catastrophic forgetting, the abrupt loss of previously acquired knowledge upon learning new information, remains the central challenge in Continual Learning.
arXiv Machine Learning
Jun 25
The Gentle Collapse: Distributional Metrics for Continual Learning
arXiv:2606. 25165v1 Announce Type: new Abstract: Accuracy degradation is the standard metric for Catastrophic Forgetting (CF), however, it records only whether forgetting occurred or not.
arXiv Machine Learning
Jun 5
Catastrophic Forgetting as Accessibility Collapse: A Three-Level Framework for Knowledge Persistence in Continual Learning
arXiv:2606. 06032v1 Announce Type: new Abstract: Catastrophic forgetting is commonly interpreted as the irreversible erasure of previously acquired knowledge during sequential learning.
Hugging Face Trending Papers
Jun 23
The Gentle Collapse: Distributional Metrics for Continual Learning
Accuracy degradation is the standard metric for Catastrophic Forgetting (CF), however, it records only whether forgetting occurred or not. It saturates at the extremes and collapses discretely at task boundaries, hiding the internal structure of what is being forgotten.
arXiv Machine Learning
Aug 4
(How) Learning Rates Regulate Catastrophic Overtraining
arXiv:2604. 13627v2 Announce Type: replace Abstract: Supervised fine-tuning (SFT) is a common first stage of LLM post-training, teaching the model to follow instructions and shaping its behavior as a helpful assistant.
arXiv Machine Learning
Aug 18
Metaplasticity as adaptive gradient preconditioning for incremental learning
arXiv:2608. 14634v1 Announce Type: new Abstract: Biological intelligence naturally prevents catastrophic forgetting through Complementary Learning Systems (CLS) theory, a macroscopic consolidation process driven at the local level by synaptic metaplasticity: the continuous, history-dependent neuromodulation of individual synapses.
arXiv Computation and Language
Oct 8
A Deafening Silence: Catastrophic Forgetting Lives in the Output Embeddings of Tokens the Data Never Speaks
arXiv:2610.09835v1 Announce Type: new Abstract: Continual pre-training and fine-tuning in Large Language Models (LLMs) inevitably induce catastrophic forgetting, typically mitigated by replay using o...