arXiv:2606. 13767v1 Announce Type: cross Abstract: Low-rank adaptation (LoRA) and its variants provide a memory- and compute-efficient alternative to full fine-tuning of pre-trained models.
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.
The paper introduces ISO-LoRA, an optimizer that improves rank utilization in Low‑Rank Adaptation (LoRA) by coupling factor updates through spectral descent on the induced tangent perturbation in weight space. Experiments on GPT‑2 adaptation show that standard optimizers like AdamW concentrate updates in a few singular directions, whereas ISO-LoRA distributes energy more evenly, leading to higher effective rank and better downstream performance across 0.1B‑7B models. The authors provide theoretical guarantees under a stylized spiked‑gradient model and demonstrate that ISO-LoRA consistently outperforms factor‑wise optimizers, especially at moderate‑to‑large LoRA ranks.
arXiv:2607. 26247v1 Announce Type: new Abstract: Low-rank adaptation (LoRA) fine-tunes large pretrained models at a fraction of the cost of full fine-tuning, but its performance depends strongly on how the adapters are initialized.
arXiv:2605.07111v3 Announce Type: replace-cross Abstract: Recent literature on fine-tuning Large Language Models highlights a fundamental debate. While Full Fine-Tuning (FFT) provides greater represe...
arXiv:2602. 05988v2 Announce Type: replace Abstract: Pre-training Large Language Models (LLMs) on web-scale datasets becomes fundamental for advancing general-purpose AI.
LoRA-TSD introduces a new optimizer for low‑rank adaptation (LoRA) that treats each update as a tangent vector on the fixed‑rank matrix manifold and applies a Muon‑style spectral‑norm steepest‑descent step within that tangent space. The method avoids costly full‑matrix operations and offers a retraction that is up to 2.8× cheaper than previous manifold approaches. The authors prove that their surrogate recovers LoRA‑Pro, identify the Riemannian gradient as the natural stationarity measure, and provide the first global convergence guarantees for both LoRA‑Pro and LoRA‑TSD, achieving superior performance across multiple benchmarks with Llama and Qwen models.
arXiv:2609.37027v1 Announce Type: new Abstract: Low-Rank Adaptation (LoRA) is a widely used approach to parameter-efficient fine-tuning (PEFT), yet a performance gap can remain relative to full fine-...