The Flow has not summarised this story yet — read it at arXiv Machine Learning.
The paper introduces LoDA, a LoRA-based continual learning method that decomposes update spaces into task‑shared and truly task‑specific subspaces using energy‑based objectives. By fixing down‑projections and learning up‑projections with Gradient‑Aligned Optimization, LoDA recalibrates general updates in closed form after each task, aiming to improve knowledge sharing and isolation. Experiments show that LoDA outperforms existing CL methods.
arXiv:2607. 09202v1 Announce Type: cross Abstract: Continual learning commonly relies on post-hoc mechanisms such as replay, elastic regularization, or distillation.
arXiv:2411. 16073v4 Announce Type: replace-cross Abstract: Inspired by the Well-initialized Lottery Ticket Hypothesis (WLTH), we introduce Soft-TransFormers (Soft-TF), a continual learning framework that adapts a frozen pre-trained Transformer through task-specific soft subnetworks: real-valued multiplicative masks over the query, key, value, and output projections of selected self-attention layers.
MuLoRA introduces a spectrally balanced low‑rank adaptation method for continual learning, addressing the issue of spectral plasticity collapse where adaptation energy concentrates in few singular modes. It uses historical whitening to identify task‑adaptive input directions and approximate polar orthogonalization of momentum updates to reduce spectral concentration, ensuring effective utilization of the low‑rank space. Experiments on five class‑incremental benchmarks and eight incremental settings show MuLoRA achieving the highest mean accuracy in 15 of 16 metrics.
arXiv:2605. 19145v3 Announce Type: replace Abstract: In the literature, many continual learning (CL) algorithms have been proposed to address the issue of catastrophic forgetting in ML models (i.
The paper introduces an algorithmic stability framework to analyze catastrophic forgetting in continual learning using SGD, without assuming a specific model architecture. It decomposes forgetting into intra‑task and cross‑task generalization gaps plus empirical forgetting, and derives bounds for different learning‑rate schedules. Experiments on Split CIFAR‑100 and Split MiniImageNet illustrate the theoretical insights.