arXiv:2608. 19800v1 Announce Type: cross Abstract: Low-Rank Adaptation (LoRA) is a prominent fine-tuning method for large models, achieving competitive performance with reduced memory overhead.
arXiv:2609.37027v1 Announce Type: new Abstract: Low-Rank Adaptation (LoRA) is a widely used approach to parameter-efficient fine-tuning (PEFT), yet a performance gap can remain relative to full fine-...
The paper introduces ISO-LoRA, an optimizer that improves rank utilization in Low‑Rank Adaptation (LoRA) by coupling factor updates through spectral descent on the induced tangent perturbation in weight space. Experiments on GPT‑2 adaptation show that standard optimizers like AdamW concentrate updates in a few singular directions, whereas ISO-LoRA distributes energy more evenly, leading to higher effective rank and better downstream performance across 0.1B‑7B models. The authors provide theoretical guarantees under a stylized spiked‑gradient model and demonstrate that ISO-LoRA consistently outperforms factor‑wise optimizers, especially at moderate‑to‑large LoRA ranks.
arXiv:2607. 11940v1 Announce Type: cross Abstract: As the scale of large pre-trained models continues to grow, fine-tuning them under limited memory budgets has become increasingly challenging.
TaRA: Training-Aware Low-Rank Adaptation Initialization proposes a new way to initialize LoRA by aligning the gradients of low‑rank factors with those of the full‑rank weight matrix. This approach directly incorporates training dynamics, improving gradient fidelity at the start of fine‑tuning while adding negligible computational cost. Experiments on a variety of challenging fine‑tuning tasks show that TaRA consistently outperforms existing state‑of‑the‑art initialization methods, offering a simple, robust, and scalable solution for effective LoRA initialization.
arXiv:2607. 26247v1 Announce Type: new Abstract: Low-rank adaptation (LoRA) fine-tunes large pretrained models at a fraction of the cost of full fine-tuning, but its performance depends strongly on how the adapters are initialized.
arXiv:2510. 24561v3 Announce Type: replace-cross Abstract: LoRA has become a widely adopted method for PEFT, and its initialization methods have attracted increasing attention.
arXiv:2606. 16454v1 Announce Type: cross Abstract: Low-Rank Adaptation (LoRA) enables efficient adaptation of large pre-trained models to downstream tasks by parameterizing weight updates with low-rank matrices.
arXiv:2607. 09757v1 Announce Type: cross Abstract: Low-Rank Adaptation (LoRA) has become a cornerstone of parameter-efficient fine-tuning (PEFT); however, the conventional practice of uniform rank assignment ignores the functional heterogeneity of neural layers.
arXiv:2606. 13767v1 Announce Type: cross Abstract: Low-rank adaptation (LoRA) and its variants provide a memory- and compute-efficient alternative to full fine-tuning of pre-trained models.
arXiv:2605.07111v3 Announce Type: replace-cross Abstract: Recent literature on fine-tuning Large Language Models highlights a fundamental debate. While Full Fine-Tuning (FFT) provides greater represe...
The paper introduces a system that enables first‑order LoRA training on phone NPUs by running all frozen‑weight multiplications on the NPU while keeping adapters and the optimizer on the CPU. It reuses weight tiles for input‑gradient computation, selects precision based on gradient sensitivity, and fuses remaining CPU operations to overlap with NPU execution. The approach is evaluated on Qualcomm Snapdragon devices with Qwen and Llama LLMs, demonstrating background training for nightly personalization and benchmark workloads.
arXiv:2601. 16991v3 Announce Type: replace-cross Abstract: Adapting large pre-trained language models to downstream tasks often entails fine-tuning millions of parameters or deploying costly dense weight updates, which hinders their use in resource-constrained environments.