arXiv:2603. 11201v3 Announce Type: replace-cross Abstract: The world is inherently dynamic, and continual learning aims to enable models to adapt to ever-evolving data streams.
By Haihua Luo, Xuming Ran, Tommi K\"arkk\"ainen, Huiyan Xue, Zhonghua Chen, Qi Xu, Fengyu Cong
arXiv:2607. 23837v1 Announce Type: new Abstract: Large language models generalize well to individual tasks but lack an inherent mechanism for learning them sequentially, leading to catastrophic forgetting.
By Reza Rahimi Azghan, Gautham Krishna Gudur, Giulia Pedrielli, Pavan Turaga, Hassan Ghasemzadeh
The paper introduces EoupCT, a framework that estimates and orthogonalizes unknown pre‑training gradients to mitigate catastrophic forgetting during continual fine‑tuning of large language models. It generates pseudo data most susceptible to forgetting using a learnable soft prompt with Gumbel‑Softmax, then jointly optimizes model parameters and the prompt via a first‑order Pareto optimizer to enforce orthogonality between new task updates and the estimated gradients. Experiments on multiple LLMs show that EoupCT preserves both task‑specific performance and the models’ inherent general‑purpose knowledge.
By Bing Wang, Changchun Li, Xin-Qiang Cai, Lin Yuanbo Wu, Ximing Li, Gang Niu, Masashi Sugiyama
ChainLoRA is a replay‑free continual learning framework for large language models that merges task vectors while preserving geometry. It uses chain‑updated training with a one‑sided orthogonality proxy to keep historical state and regularization overhead constant, and applies post‑stream adaptive SVD merging with Procrustes adaptation to separate shared and task‑specific components. Experiments demonstrate state‑of‑the‑art performance on Large and SuperNI benchmarks and competitive results on Standard CL, approaching the scores of replay‑based methods.
By Hang Yin, Haozhe Wang, Yuhua Luo, Zhangqi Pan, Xiaoxing Wang, Junchi Yan
CRAFT is a continual learning framework that freezes model weights and learns low‑rank interventions on hidden representations to mitigate catastrophic forgetting. Tasks that are similar share an intervention, determined by how they change the model’s output distribution, and a KL penalty keeps new tasks close to existing behavior. The approach also allows task identification at test time without explicit labels and shows improved performance and reduced forgetting across benchmarks compared to LoRA‑based methods.
By Md Anwar Hossen, Fatema Siddika, Juan Pablo Munoz, Tanya Roosta, Ali Jannesari
arXiv:2606. 07500v1 Announce Type: cross Abstract: Continual learning in Large Language Models (LLMs) is hindered by the plasticity-stability dilemma, where acquiring new capabilities often leads to catastrophic forgetting of previous knowledge.
By Fatema Siddika, Md Anwar Hossen, Tanwi Mallick, Ali Jannesari