← Back to all news
arXiv Machine Learning By Guodong Zheng, Yan Sun, Peng Wang, Enneng Yang, Jiafei Lyu, Xiaochun Cao, Dacheng Tao, Li Shen

Fine-Grained Analysis of Catastrophic Forgetting in Continual Learning via Algorithmic Stability

Read the original on arXiv Machine Learning →

The paper introduces an algorithmic stability framework to analyze catastrophic forgetting in continual learning using SGD, without assuming a specific model architecture. It decomposes forgetting into intra‑task and cross‑task generalization gaps plus empirical forgetting, and derives bounds for different learning‑rate schedules. Experiments on Split CIFAR‑100 and Split MiniImageNet illustrate the theoretical insights.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
Sep 28

Estimating and Orthogonalizing Unknown Pre-training Gradients for Continual Fine-tuning of Large Language Models

The paper introduces EoupCT, a framework that estimates and orthogonalizes unknown pre‑training gradients to mitigate catastrophic forgetting during continual fine‑tuning of large language models. It generates pseudo data most susceptible to forgetting using a learnable soft prompt with Gumbel‑Softmax, then jointly optimizes model parameters and the prompt via a first‑order Pareto optimizer to enforce orthogonality between new task updates and the estimated gradients. Experiments on multiple LLMs show that EoupCT preserves both task‑specific performance and the models’ inherent general‑purpose knowledge.

By Bing Wang, Changchun Li, Xin-Qiang Cai, Lin Yuanbo Wu, Ximing Li, Gang Niu, Masashi Sugiyama
llmsfine-tuning
More like this →