arXiv:2603. 12658v2 Announce Type: replace-cross Abstract: Continual learning (CL) has emerged as a pivotal paradigm to enable large language models (LLMs) to dynamically adapt to evolving knowledge and sequential tasks while mitigating catastrophic forgetting, a critical limitation of the static pre-training paradigm inherent to modern LLMs.
By Hongyang Chen, Zhongwu Sun, Hongfei Ye, Kunchi Li, Xuemin Lin
The paper introduces EoupCT, a framework that estimates and orthogonalizes unknown pre‑training gradients to mitigate catastrophic forgetting during continual fine‑tuning of large language models. It generates pseudo data most susceptible to forgetting using a learnable soft prompt with Gumbel‑Softmax, then jointly optimizes model parameters and the prompt via a first‑order Pareto optimizer to enforce orthogonality between new task updates and the estimated gradients. Experiments on multiple LLMs show that EoupCT preserves both task‑specific performance and the models’ inherent general‑purpose knowledge.
By Bing Wang, Changchun Li, Xin-Qiang Cai, Lin Yuanbo Wu, Ximing Li, Gang Niu, Masashi Sugiyama
arXiv:2609.06986v1 Announce Type: new
Abstract: Language models may need to internalize information that arrives over time and retain it through many subsequent updates. To study this challenge, we i...
By Zheyuan Zhang, Alvin Zhang, Daniel Khashabi, Tianmin Shu
The paper introduces PIECE, a Parameter Importance-Driven Continual Learning method that selectively updates only 0.1% of core parameters to preserve general abilities while learning new domain knowledge. PIECE employs two importance estimators—PIECE‑F using Fisher Information and PIECE‑S combining gradient and curvature information—to guide updates. Experiments on three language models and two multimodal models demonstrate that PIECE maintains general capabilities and achieves state‑of‑the‑art continual learning performance without accessing prior training data or adding parameter overhead.
By Lingxiang Wang, Hainan Zhang, Zhiming Zheng
arXiv:2504. 13822v3 Announce Type: replace-cross Abstract: The emergence of large pre-trained networks has revolutionized the AI field, unlocking new possibilities and achieving unprecedented performance.
By Eric Nuertey Coleman, Luigi Quarantiello, Ziyue Liu, Qinwen Yang, Samrat Mukherjee, Julio Hurtado, Vincenzo Lomonaco
arXiv:2603. 11201v3 Announce Type: replace-cross Abstract: The world is inherently dynamic, and continual learning aims to enable models to adapt to ever-evolving data streams.
By Haihua Luo, Xuming Ran, Tommi K\"arkk\"ainen, Huiyan Xue, Zhonghua Chen, Qi Xu, Fengyu Cong