arXiv Machine Learning October 8, 2026 By Yikuan Li, Pinyan Lu, Fanghui Liu

Are Parameter-Efficient Fine-tuning Methods Really Different?

Read the original on arXiv Machine Learning →

The paper investigates whether different parameter‑efficient fine‑tuning (PEFT) methods truly differ in their approach and impact. By comparing six methods on language and diffusion models, the authors find that many LoRA‑family techniques approximately preserve pretrained geometry, and that restoring slightly drifted singular‑value spectra largely maintains task performance, challenging the need for explicit geometric preservation. They also observe distinct adaptation–retention trade‑offs: LoRA limits forgetting best, DoRA scores higher on average, and PiSSA incurs more retention costs, with performance gains linked to modifications in specific spectral components.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv AI

Oct 1

Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation

arXiv:2605.07111v3 Announce Type: replace-cross Abstract: Recent literature on fine-tuning Large Language Models highlights a fundamental debate. While Full Fine-Tuning (FFT) provides greater represe...

By Haozhan Tang, Xiuqi Zhu, Xinyin Zhang, Boxun Li, Virginia Smith, Kevin Kuo

llms fine-tuning data-engineering

arXiv Machine Learning

Sep 1

Least but not Last: Fine-tuning Intermediate Principal Components for Better Performance-Forgetting Trade-Offs

arXiv:2602.03493v2 Announce Type: replace Abstract: Low-Rank Adaptation (LoRA) methods have emerged as crucial techniques for adapting large pre-trained models to downstream tasks under computational...

By Alessio Quercia, Arya Bangun, Ira Assent, Hanno Scharr

computer-vision nlp fine-tuning

arXiv Machine Learning

Jul 23

The Blessing of Dimensionality: How Near-Orthogonality in High-Dimensional Spaces Explains Temporal Portability

arXiv:2607. 20301v1 Announce Type: new Abstract: Fine-tuning has been widely used to adapt large language models (LLMs) for domain-specific tasks.

By Abigail Woodring, Adrian Chan, Rana Muhammad Shahroz Khan, Sukwon Yun, Chau-Wai Wong, Tianlong Chen

llms fine-tuning

arXiv Machine Learning

Oct 5

MuLoRA: Spectrally Balanced Low-Rank Adaptation for Continual Learning

MuLoRA introduces a spectrally balanced low‑rank adaptation method for continual learning, addressing the issue of spectral plasticity collapse where adaptation energy concentrates in few singular modes. It uses historical whitening to identify task‑adaptive input directions and approximate polar orthogonalization of momentum updates to reduce spectral concentration, ensuring effective utilization of the low‑rank space. Experiments on five class‑incremental benchmarks and eight incremental settings show MuLoRA achieving the highest mean accuracy in 15 of 16 metrics.

By Junkang Liu

fine-tuning benchmarks

arXiv AI

Sep 30

Beyond Low-Rank Parameterization: Narrowing the Gap Between LoRA and Full Fine-Tuning via Gradient Decomposition

arXiv:2609.37027v1 Announce Type: new Abstract: Low-Rank Adaptation (LoRA) is a widely used approach to parameter-efficient fine-tuning (PEFT), yet a performance gap can remain relative to full fine-...

By Yihao Ouyang, Shiwei Li, Haozhao Wang, Xiandi Luo, Zhuoqi Hu, Jinglun Yu, Yichen Li, Ruixuan Li

computer-vision fine-tuning

arXiv Machine Learning

Jun 9

Shortcuts in the Tail: Debiasing via Post-Hoc Spectral Compression of Fine-Tuning Updates

arXiv:2606. 07596v1 Announce Type: new Abstract: Fine-tuning often introduces spurious correlations alongside task knowledge, causing systematic failures on underrepresented groups.

By Edward Sun, Dmitrii Troitskii

fine-tuning benchmarks

One email a morning, machine-written

Related stories

Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation

Least but not Last: Fine-tuning Intermediate Principal Components for Better Performance-Forgetting Trade-Offs

The Blessing of Dimensionality: How Near-Orthogonality in High-Dimensional Spaces Explains Temporal Portability

MuLoRA: Spectrally Balanced Low-Rank Adaptation for Continual Learning

Beyond Low-Rank Parameterization: Narrowing the Gap Between LoRA and Full Fine-Tuning via Gradient Decomposition

Shortcuts in the Tail: Debiasing via Post-Hoc Spectral Compression of Fine-Tuning Updates