Normalized Low-Rank Adaptation (NoRA) is a lightweight enhancement to the widely used LoRA technique that normalizes the down‑projection matrices during training. By doing so, NoRA stabilizes early optimization dynamics, accelerates convergence, and improves performance across pretraining, supervised fine‑tuning, and reinforcement learning. The method adds no extra trainable parameters or inference‑time cost, making it broadly applicable.
By Jiale Kang, Ziyin Yue, Zheng Zhan, Yangyi Huang, Weiyang Liu
arXiv:2510. 24561v3 Announce Type: replace-cross Abstract: LoRA has become a widely adopted method for PEFT, and its initialization methods have attracted increasing attention.
By Qingyue Zhang, Chang Chu, Tianren Peng, Qi Li, Xiangyang Luo, Zhihao Jiang, Shao-Lun Huang
arXiv:2606. 16454v1 Announce Type: cross Abstract: Low-Rank Adaptation (LoRA) enables efficient adaptation of large pre-trained models to downstream tasks by parameterizing weight updates with low-rank matrices.
By Junghun Oh, Sungyong Baik, Kyoung Mu Lee
arXiv:2609.37027v1 Announce Type: new
Abstract: Low-Rank Adaptation (LoRA) is a widely used approach to parameter-efficient fine-tuning (PEFT), yet a performance gap can remain relative to full fine-...
By Yihao Ouyang, Shiwei Li, Haozhao Wang, Xiandi Luo, Zhuoqi Hu, Jinglun Yu, Yichen Li, Ruixuan Li
arXiv:2608. 19800v1 Announce Type: cross Abstract: Low-Rank Adaptation (LoRA) is a prominent fine-tuning method for large models, achieving competitive performance with reduced memory overhead.
By Haonan He, Xinyue Fan
The paper introduces Activation Boundary Matching for Low‑Rank Adaptation (ABM‑LoRA), a task‑informed initialization strategy that uses the signs of layer‑wise pre‑activations from a brief probe adapter as targets for a fresh adapter. By training with a margin‑based hinge objective on these activation boundaries, ABM‑LoRA captures useful adaptation directions that standard LoRA initializers miss, while requiring only a few forward passes. Experiments show that ABM‑LoRA outperforms or matches existing LoRA, SVD, and gradient‑based initializers across multiple models and benchmarks, including T5‑base/GLUE, ConvNeXt‑T, Swin‑T, Qwen2.5‑1.5B, and LLaMA2‑7B.
By Dongha Lee, Jinhee Park, Minjun Kim, Junseok Kwon
arXiv:2607. 11940v1 Announce Type: cross Abstract: As the scale of large pre-trained models continues to grow, fine-tuning them under limited memory budgets has become increasingly challenging.
By Gengyu Zhang, Haiyin Ran, Zhengbao He, Yuhang Liu, Hanling Tian, Zhehao Huang, Xiaolin Huang
arXiv:2607. 09757v1 Announce Type: cross Abstract: Low-Rank Adaptation (LoRA) has become a cornerstone of parameter-efficient fine-tuning (PEFT); however, the conventional practice of uniform rank assignment ignores the functional heterogeneity of neural layers.
By Jiaqi Liu, Haidong Kang, Qihui Zhao, Guo Yu
The paper introduces ISO-LoRA, an optimizer that improves rank utilization in Low‑Rank Adaptation (LoRA) by coupling factor updates through spectral descent on the induced tangent perturbation in weight space. Experiments on GPT‑2 adaptation show that standard optimizers like AdamW concentrate updates in a few singular directions, whereas ISO-LoRA distributes energy more evenly, leading to higher effective rank and better downstream performance across 0.1B‑7B models. The authors provide theoretical guarantees under a stylized spiked‑gradient model and demonstrate that ISO-LoRA consistently outperforms factor‑wise optimizers, especially at moderate‑to‑large LoRA ranks.
By Zihan Zhu, Zhehang Du, Xuyang Chen, Tim Tsz-Kit Lau, Jiayuan Wu, X. Y. Han, Qi Long, Weijie Su
arXiv:2607. 19391v1 Announce Type: cross Abstract: Low-Rank Adaptation is widely used for parameter-efficient fine-tuning, yet existing methods typically assign the same adapter rank to every transformer layer despite their heterogeneous adaptation requirements.
By Ashutosh Tripathi, Surya Deep Singh, Pranab Sahoo, Sriparna Saha
arXiv:2610.23343v1 Announce Type: new
Abstract: Low-rank adaptation (LoRA) has become the standard method for parameter-efficient fine-tuning of large pretrained models, yet theoretical explanations...
By Long Nguyen-Chi, Quynh Nguyen, Thanh Nguyen-Cung, Binh T. Nguyen
arXiv:2610.19700v1 Announce Type: new
Abstract: Low-rank adaptation (LoRA) enables parameter-efficient fine-tuning by learning low-rank weight updates. Increasing rank expands adaptation capacity, bu...
By Fanyao Li, Yuxi Liu, Xiaohao Liu, He Chang, Zhaoxin Wang, Zhulin Tao