The paper introduces Subspace Tuning (SST), a fine‑tuning technique that restricts weight updates to the bottom singular subspace of model parameters, thereby protecting the principal directions that are most prone to catastrophic forgetting. SST is optimizer‑agnostic, works with existing fine‑tuning recipes such as SFT and self‑distillation, and includes dense gradient estimation and a refresh mechanism to enhance learning. Experiments show that SST can preserve up to 99 % of the weight subspace, significantly reduce KL drift, and nearly eliminate forgetting in large‑scale language models while maintaining strong target‑task performance.
By Arnav Chavan, Kusha Sareen, Rishabh Tiwari, Abhranil Chandra, Johan Obando-Ceron, Kurt Keutzer, Scott Niekum, Inderjit S Dhillon, Devvrit Khatri
arXiv:2603. 12658v2 Announce Type: replace-cross Abstract: Continual learning (CL) has emerged as a pivotal paradigm to enable large language models (LLMs) to dynamically adapt to evolving knowledge and sequential tasks while mitigating catastrophic forgetting, a critical limitation of the static pre-training paradigm inherent to modern LLMs.
By Hongyang Chen, Zhongwu Sun, Hongfei Ye, Kunchi Li, Xuemin Lin
arXiv:2605. 05285v2 Announce Type: replace Abstract: Large language models (LLMs) often suffer from catastrophic forgetting in continual learning: after learning new tasks sequentially, they perform worse on earlier tasks.
By Yazheng Liu, Yuxuan Wan, Rui Xu, Xi Zhang, Sihong Xie, Hui Xiong
arXiv:2610.09835v1 Announce Type: new
Abstract: Continual pre-training and fine-tuning in Large Language Models (LLMs) inevitably induce catastrophic forgetting, typically mitigated by replay using o...
By Jonghyun Han, Younghoon Song, Jongyoul Park
The paper introduces an algorithmic stability framework to analyze catastrophic forgetting in continual learning using SGD, without assuming a specific model architecture. It decomposes forgetting into intra‑task and cross‑task generalization gaps plus empirical forgetting, and derives bounds for different learning‑rate schedules. Experiments on Split CIFAR‑100 and Split MiniImageNet illustrate the theoretical insights.
By Guodong Zheng, Yan Sun, Peng Wang, Enneng Yang, Jiafei Lyu, Xiaochun Cao, Dacheng Tao, Li Shen
arXiv:2606. 07500v1 Announce Type: cross Abstract: Continual learning in Large Language Models (LLMs) is hindered by the plasticity-stability dilemma, where acquiring new capabilities often leads to catastrophic forgetting of previous knowledge.
By Fatema Siddika, Md Anwar Hossen, Tanwi Mallick, Ali Jannesari