arXiv Machine Learning August 14, 2026 By Hyowon Wi, Noseong Park

Can Spectral-Clipping Enable Better Learning While Forgetting Less for Low-Rank Adaptation?

Read the original on arXiv Machine Learning →

arXiv:2608. 12332v1 Announce Type: cross Abstract: In recent years, low-rank adaptation (LoRA) has emerged as a significant paradigm that freezes pre-trained weights and introduces small, learnable adapters instead of fine-tuning the full set of parameters.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv Machine Learning.

fine-tuning

arXiv Machine Learning

Jul 3

Spectral Imbalance Causes Forgetting in Low-Rank Continual Adaptation

arXiv:2602. 00722v2 Announce Type: replace Abstract: Parameter-efficient continual learning aims to adapt pre-trained models to sequential tasks without forgetting previously acquired knowledge.

By Hao Gu, Mao-Lin Luo, Zi-Hao Zhou, Han-Chen Zhang, Min-Ling Zhang, Tong Wei

llms multimodal

arXiv AI

Jun 16

SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation

arXiv:2606. 16454v1 Announce Type: cross Abstract: Low-Rank Adaptation (LoRA) enables efficient adaptation of large pre-trained models to downstream tasks by parameterizing weight updates with low-rank matrices.

By Junghun Oh, Sungyong Baik, Kyoung Mu Lee

fine-tuning benchmarks safety

arXiv AI

Jun 2

Foundation-Preserving Adaptation via Generalized Rayleigh-Quotient Optimization

arXiv:2606. 00132v1 Announce Type: cross Abstract: While finetuning effectively adapts foundation models to specialized downstream tasks, it can degrade nontarget capabilities acquired during pretraining.

By Dongjun Kim, Adrian de Wynter, Huancheng Chen, Heasung Kim, Haris Vikalo

fine-tuning

arXiv AI

Jul 20

PASs-MoE: Mitigating Misaligned Co-drift among Router and Experts via Pathway Activation Subspaces for Continual Learning

arXiv:2601. 13020v2 Announce Type: replace-cross Abstract: Continual instruction tuning (CIT) requires multimodal large language models (MLLMs) to adapt to a stream of tasks without forgetting prior capabilities.

By Zhiyan Hou, Haiyun Guo, Haokai Ma, Yandu Sun, Yonghui Yang, Jinqiao Wang

llms fine-tuning multimodal benchmarks

arXiv AI

Jun 17

Catastrophic Forgetting is Low-Rank: A Function-Space Theory for Continual Adaptation

arXiv:2606. 18024v1 Announce Type: cross Abstract: Catastrophic forgetting in continual adaptation is usually studied through parameter drift, replay, or distillation, but these views do not identify which output-space directions are vulnerable.

By Ido Nitzan Hidekel, Dan Raviv

fine-tuning efficiency

arXiv Machine Learning

Jun 29

When One Adapter Speaks for Many: Discovering Low-Rank Redundancy in Continual Fine-Tuning

arXiv:2606. 28117v1 Announce Type: new Abstract: Low-Rank Adaptation (LoRA) has become the standard tool for parameter-efficient fine-tuning of large pretrained models.

By Tanguy Dieudonn\'e, Giulia Lanzillotta, Enis Simsar, Louis Barinka, Thomas Hofmann

fine-tuning benchmarks

Related stories

Spectral Imbalance Causes Forgetting in Low-Rank Continual Adaptation

SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation

Foundation-Preserving Adaptation via Generalized Rayleigh-Quotient Optimization

PASs-MoE: Mitigating Misaligned Co-drift among Router and Experts via Pathway Activation Subspaces for Continual Learning

Catastrophic Forgetting is Low-Rank: A Function-Space Theory for Continual Adaptation

When One Adapter Speaks for Many: Discovering Low-Rank Redundancy in Continual Fine-Tuning