The paper investigates whether different parameter‑efficient fine‑tuning (PEFT) methods truly differ in their approach and impact. By comparing six methods on language and diffusion models, the authors find that many LoRA‑family techniques approximately preserve pretrained geometry, and that restoring slightly drifted singular‑value spectra largely maintains task performance, challenging the need for explicit geometric preservation. They also observe distinct adaptation–retention trade‑offs: LoRA limits forgetting best, DoRA scores higher on average, and PiSSA incurs more retention costs, with performance gains linked to modifications in specific spectral components.
arXiv:2605.07111v3 Announce Type: replace-cross Abstract: Recent literature on fine-tuning Large Language Models highlights a fundamental debate. While Full Fine-Tuning (FFT) provides greater represe...
arXiv:2602.03493v2 Announce Type: replace Abstract: Low-Rank Adaptation (LoRA) methods have emerged as crucial techniques for adapting large pre-trained models to downstream tasks under computational...
arXiv:2607. 20301v1 Announce Type: new Abstract: Fine-tuning has been widely used to adapt large language models (LLMs) for domain-specific tasks.
MuLoRA introduces a spectrally balanced low‑rank adaptation method for continual learning, addressing the issue of spectral plasticity collapse where adaptation energy concentrates in few singular modes. It uses historical whitening to identify task‑adaptive input directions and approximate polar orthogonalization of momentum updates to reduce spectral concentration, ensuring effective utilization of the low‑rank space. Experiments on five class‑incremental benchmarks and eight incremental settings show MuLoRA achieving the highest mean accuracy in 15 of 16 metrics.
arXiv:2609.37027v1 Announce Type: new Abstract: Low-Rank Adaptation (LoRA) is a widely used approach to parameter-efficient fine-tuning (PEFT), yet a performance gap can remain relative to full fine-...
arXiv:2606. 07596v1 Announce Type: new Abstract: Fine-tuning often introduces spurious correlations alongside task knowledge, causing systematic failures on underrepresented groups.
The paper introduces ISO-LoRA, an optimizer that improves rank utilization in Low‑Rank Adaptation (LoRA) by coupling factor updates through spectral descent on the induced tangent perturbation in weight space. Experiments on GPT‑2 adaptation show that standard optimizers like AdamW concentrate updates in a few singular directions, whereas ISO-LoRA distributes energy more evenly, leading to higher effective rank and better downstream performance across 0.1B‑7B models. The authors provide theoretical guarantees under a stylized spiked‑gradient model and demonstrate that ISO-LoRA consistently outperforms factor‑wise optimizers, especially at moderate‑to‑large LoRA ranks.
arXiv:2606. 18024v1 Announce Type: cross Abstract: Catastrophic forgetting in continual adaptation is usually studied through parameter drift, replay, or distillation, but these views do not identify which output-space directions are vulnerable.
arXiv:2601. 13020v2 Announce Type: replace-cross Abstract: Continual instruction tuning (CIT) requires multimodal large language models (MLLMs) to adapt to a stream of tasks without forgetting prior capabilities.
arXiv:2606. 13767v1 Announce Type: cross Abstract: Low-rank adaptation (LoRA) and its variants provide a memory- and compute-efficient alternative to full fine-tuning of pre-trained models.
arXiv:2607. 07719v1 Announce Type: cross Abstract: Parameter-efficient fine-tuning adapts a large language model to one task cheaply, but across a task sequence LoRA-style methods keep stacking low-rank updates on the same frozen weight, so each new task tends to overwrite the previous ones.
arXiv:2606. 06920v1 Announce Type: cross Abstract: Deploying Small Language Models (SLMs) on edge devices requires efficient fine-tuning strategies that adapt models to new tasks without degrading their general capabilities.