The paper introduces LoDA, a LoRA-based continual learning method that decomposes update spaces into task‑shared and truly task‑specific subspaces using energy‑based objectives. By fixing down‑projections and learning up‑projections with Gradient‑Aligned Optimization, LoDA recalibrates general updates in closed form after each task, aiming to improve knowledge sharing and isolation. Experiments show that LoDA outperforms existing CL methods.
By Lingfeng He, De Cheng, Huaijie Wang, Xi Yang, Nannan Wang, Xinbo Gao
arXiv:2607. 09202v1 Announce Type: cross Abstract: Continual learning commonly relies on post-hoc mechanisms such as replay, elastic regularization, or distillation.
By Julius St\"ork
arXiv:2411. 16073v4 Announce Type: replace-cross Abstract: Inspired by the Well-initialized Lottery Ticket Hypothesis (WLTH), we introduce Soft-TransFormers (Soft-TF), a continual learning framework that adapts a frozen pre-trained Transformer through task-specific soft subnetworks: real-valued multiplicative masks over the query, key, value, and output projections of selected self-attention layers.
By Haeyong Kang, Chang D. Yoo
MuLoRA introduces a spectrally balanced low‑rank adaptation method for continual learning, addressing the issue of spectral plasticity collapse where adaptation energy concentrates in few singular modes. It uses historical whitening to identify task‑adaptive input directions and approximate polar orthogonalization of momentum updates to reduce spectral concentration, ensuring effective utilization of the low‑rank space. Experiments on five class‑incremental benchmarks and eight incremental settings show MuLoRA achieving the highest mean accuracy in 15 of 16 metrics.
By Junkang Liu
arXiv:2605. 19145v3 Announce Type: replace Abstract: In the literature, many continual learning (CL) algorithms have been proposed to address the issue of catastrophic forgetting in ML models (i.
By Srijith Nair, Atilla Eryilmaz, Jia Liu
The paper introduces an algorithmic stability framework to analyze catastrophic forgetting in continual learning using SGD, without assuming a specific model architecture. It decomposes forgetting into intra‑task and cross‑task generalization gaps plus empirical forgetting, and derives bounds for different learning‑rate schedules. Experiments on Split CIFAR‑100 and Split MiniImageNet illustrate the theoretical insights.
By Guodong Zheng, Yan Sun, Peng Wang, Enneng Yang, Jiafei Lyu, Xiaochun Cao, Dacheng Tao, Li Shen
arXiv:2609.24678v1 Announce Type: new
Abstract: Continual learning with Low-Rank Adapters (LoRA) typically mitigates forgetting by penalizing the overlap between a new update and the accumulated past...
By Sebastian George Sincari (Faculty of Mathematics and Computer Science, University of Bucharest, Bucharest, Romania), Bogdan Alexandru Gheorghe (Faculty of Mathematics and Computer Science, University of Bucharest, Bucharest, Romania), Antonio Barbalau (Bitdefender, Bucharest, Romania)
arXiv:2608.31096v1 Announce Type: cross
Abstract: Class-incremental learning (CIL) requires a model to incrementally learn tasks that contain new classes without accessing earlier training data while...
By Yunxiang Fu, Meng Lou, Yizhou Yu
arXiv:2601. 13020v2 Announce Type: replace-cross Abstract: Continual instruction tuning (CIT) requires multimodal large language models (MLLMs) to adapt to a stream of tasks without forgetting prior capabilities.
By Zhiyan Hou, Haiyun Guo, Haokai Ma, Yandu Sun, Yonghui Yang, Jinqiao Wang
The paper introduces FiUni, a Fisher-guided unified framework that performs batch-level task detection and parameter-efficient continual adaptation for large language models. By exploiting orthogonality in the Fisher information matrix’s Kronecker-Factored Approximate Curvature subspaces, FiUni constructs frozen subspaces to guide low-rank adaptation and matches incoming batch windows to historical subspaces. This approach allows the model to decide whether to reuse, expand, or create new subspaces, balancing knowledge sharing and task isolation while achieving competitive performance with fewer trainable parameters.
By Dezheng Han, Anbang Zhang, Zhihao Zhu, Shuaishuai Guo
arXiv:2609.39367v1 Announce Type: new
Abstract: Despite the widespread use of Low-Rank Adaptation (LoRA), little is known about its dynamics in continual learning and the mechanisms by which low-rank...
By Th\'eo Marchetta, Filippo Alessandroni, Alessandro Breccia, Alessandro Ingrosso, Federica Gerace
The paper investigates how to add new skills to large multimodal models (LMMs) without erasing previously learned abilities. By sequentially fine‑tuning on five target skills and tracking performance on eight held‑out benchmarks across three model families, the authors discover that forgetting on one skill can partially recover when the model is later tuned on a different skill. They attribute this to shifts in the output token distribution and propose two simple, selective tuning recipes—updating only self‑attention projection layers or only the MLP Gate&Up while freezing the Down projection—that achieve a better learning‑forgiveness trade‑off than full‑LLM tuning or other common forgetting‑mitigation methods.
whyItMatters":"The study shows that carefully choosing which model components to fine‑tune can enable LMMs to acquire new skills while preserving prior knowledge, offering a simpler and more effective alternative to existing forgetting‑mitigation techniques."
By Zhen Zhu, Yiming Gong, Yao Xiao, Yaoyao Liu, Derek Hoiem