arXiv:2608. 12332v1 Announce Type: cross Abstract: In recent years, low-rank adaptation (LoRA) has emerged as a significant paradigm that freezes pre-trained weights and introduces small, learnable adapters instead of fine-tuning the full set of parameters.
The paper introduces Foundation Preserving LoRA (FoLoRA), a forgetting‑aware optimization framework that balances adaptation to downstream tasks with preservation of pretraining capabilities. FoLoRA uses a first‑order preservation condition to define a forgetting penalty based on pretraining‑proxy activations and a task utility from downstream activations, scoring update directions via a generalized Rayleigh quotient. This spectral coordinate system enables gated Adam updates that reduce low‑utility, high‑penalty directions, and the method constructs pretraining proxy calibration data by sampling from the pretrained model. Experiments on math, code, and instruction‑following tasks demonstrate that FoLoRA achieves a stronger balance between target task performance and aggregate preservation of non‑target capabilities compared to baselines.
CRAFT is a continual learning framework that freezes model weights and learns low‑rank interventions on hidden representations to mitigate catastrophic forgetting. Tasks that are similar share an intervention, determined by how they change the model’s output distribution, and a KL penalty keeps new tasks close to existing behavior. The approach also allows task identification at test time without explicit labels and shows improved performance and reduced forgetting across benchmarks compared to LoRA‑based methods.
arXiv:2602. 00722v2 Announce Type: replace Abstract: Parameter-efficient continual learning aims to adapt pre-trained models to sequential tasks without forgetting previously acquired knowledge.
The paper introduces EoupCT, a framework that estimates and orthogonalizes unknown pre‑training gradients to mitigate catastrophic forgetting during continual fine‑tuning of large language models. It generates pseudo data most susceptible to forgetting using a learnable soft prompt with Gumbel‑Softmax, then jointly optimizes model parameters and the prompt via a first‑order Pareto optimizer to enforce orthogonality between new task updates and the estimated gradients. Experiments on multiple LLMs show that EoupCT preserves both task‑specific performance and the models’ inherent general‑purpose knowledge.
C-LoRA introduces a continual learning approach for pre‑trained visual models that uses a single shared Low‑Rank Adaptation (LoRA) adapter. It employs a learnable routing matrix split into a stability component (R_base) and a plasticity component (R_delta) to balance preserving past knowledge and adapting to new tasks. This design eliminates the need for expanding modules or complex inference fusion while maintaining competitive performance on multiple benchmarks.
arXiv:2606. 00132v1 Announce Type: cross Abstract: While finetuning effectively adapts foundation models to specialized downstream tasks, it can degrade nontarget capabilities acquired during pretraining.
arXiv:2606. 28117v1 Announce Type: new Abstract: Low-Rank Adaptation (LoRA) has become the standard tool for parameter-efficient fine-tuning of large pretrained models.
arXiv:2601. 13020v2 Announce Type: replace-cross Abstract: Continual instruction tuning (CIT) requires multimodal large language models (MLLMs) to adapt to a stream of tasks without forgetting prior capabilities.
arXiv:2610.02126v1 Announce Type: cross Abstract: We explore catastrophic forgetting in the context of large pre-trained models. By considering forgetting as a geometric problem in the input space of...
The paper introduces Subspace Tuning (SST), a fine‑tuning technique that restricts weight updates to the bottom singular subspace of model parameters, thereby protecting the principal directions that are most prone to catastrophic forgetting. SST is optimizer‑agnostic, works with existing fine‑tuning recipes such as SFT and self‑distillation, and includes dense gradient estimation and a refresh mechanism to enhance learning. Experiments show that SST can preserve up to 99 % of the weight subspace, significantly reduce KL drift, and nearly eliminate forgetting in large‑scale language models while maintaining strong target‑task performance.
arXiv:2603. 11201v3 Announce Type: replace-cross Abstract: The world is inherently dynamic, and continual learning aims to enable models to adapt to ever-evolving data streams.