The paper introduces FiUni, a Fisher-guided unified framework that performs batch-level task detection and parameter-efficient continual adaptation for large language models. By exploiting orthogonality in the Fisher information matrix’s Kronecker-Factored Approximate Curvature subspaces, FiUni constructs frozen subspaces to guide low-rank adaptation and matches incoming batch windows to historical subspaces. This approach allows the model to decide whether to reuse, expand, or create new subspaces, balancing knowledge sharing and task isolation while achieving competitive performance with fewer trainable parameters.
By Dezheng Han, Anbang Zhang, Zhihao Zhu, Shuaishuai Guo
arXiv:2607. 09202v1 Announce Type: cross Abstract: Continual learning commonly relies on post-hoc mechanisms such as replay, elastic regularization, or distillation.
By Julius St\"ork
arXiv:2603. 12658v2 Announce Type: replace-cross Abstract: Continual learning (CL) has emerged as a pivotal paradigm to enable large language models (LLMs) to dynamically adapt to evolving knowledge and sequential tasks while mitigating catastrophic forgetting, a critical limitation of the static pre-training paradigm inherent to modern LLMs.
By Hongyang Chen, Zhongwu Sun, Hongfei Ye, Kunchi Li, Xuemin Lin
The paper introduces EoupCT, a framework that estimates and orthogonalizes unknown pre‑training gradients to mitigate catastrophic forgetting during continual fine‑tuning of large language models. It generates pseudo data most susceptible to forgetting using a learnable soft prompt with Gumbel‑Softmax, then jointly optimizes model parameters and the prompt via a first‑order Pareto optimizer to enforce orthogonality between new task updates and the estimated gradients. Experiments on multiple LLMs show that EoupCT preserves both task‑specific performance and the models’ inherent general‑purpose knowledge.
By Bing Wang, Changchun Li, Xin-Qiang Cai, Lin Yuanbo Wu, Ximing Li, Gang Niu, Masashi Sugiyama
arXiv:2607. 15587v1 Announce Type: new Abstract: Continual learning studies how deployed language models can continually acquire new tasks without expensive retraining from scratch.
By Yang Meng, Zhenya Liu, Zhuokai Zhao, Yuxin Chen
arXiv:2606. 01873v1 Announce Type: new Abstract: LLM-as-Aligner has emerged as a prevalent pre-training paradigm for Text-Attributed Graphs(TAGS), aligning graph and text modalities into a shared embedding space via CLIP-style contrastive learning.
By Yuhan Wang, Yibo Ding, Yutong Ye, Mufan Zhao, Wenbo Zhang, Ruijie Wang, Jianxin Li
arXiv:2609.06986v1 Announce Type: new
Abstract: Language models may need to internalize information that arrives over time and retain it through many subsequent updates. To study this challenge, we i...
By Zheyuan Zhang, Alvin Zhang, Daniel Khashabi, Tianmin Shu
arXiv:2608.31096v1 Announce Type: cross
Abstract: Class-incremental learning (CIL) requires a model to incrementally learn tasks that contain new classes without accessing earlier training data while...
By Yunxiang Fu, Meng Lou, Yizhou Yu
arXiv:2608. 11690v1 Announce Type: new Abstract: Continual learning must absorb new tasks without erasing old ones, and replay---mixing a small buffer of past examples into current training---is among the most effective remedies for catastrophic forgetting.
By Tieliang Gong, Zhongbo Zhang, Wen Wen, Yong-Jin Liu
arXiv:2607. 23837v1 Announce Type: new Abstract: Large language models generalize well to individual tasks but lack an inherent mechanism for learning them sequentially, leading to catastrophic forgetting.
By Reza Rahimi Azghan, Gautham Krishna Gudur, Giulia Pedrielli, Pavan Turaga, Hassan Ghasemzadeh
arXiv:2605. 20247v2 Announce Type: replace-cross Abstract: Catastrophic forgetting remains a major obstacle to continual learning in large language models (LLMs) and vision--language models (VLMs).
By Yang Liu, Toan Nguyen, Flora D. Salim
arXiv:2606. 02322v1 Announce Type: cross Abstract: In dynamic environments, large language models need to keep adapting to new tasks, but continual learning often suffers from forgetting, limited transfer, and vulnerability to adversarial perturbations.
By Ran Liu, Min Yu, Mingqi Liu, Jianguo Jiang, Gang Li, Rongsheng Li, Ning Li, Zhen Xu, Weiqing Huang, Ming Liu