← Back to all news
arXiv Statistics ML By Kamillo Ferry, Anthea Monod, Keisuke Yano

Singular Geometry of Low-Rank Adaptation

Read the original on arXiv Statistics ML →

The paper investigates the singular statistical nature of Low‑Rank Adaptation (LoRA), a popular parameter‑efficient fine‑tuning method for large neural networks. By applying the real log canonical threshold (RLCT) from singular learning theory, the authors analyze how LoRA’s learning complexity depends on its low‑rank factorization and the pretrained network. They show that the forward‑backward covariance, derived from activations and backpropagated sensitivities, governs which low‑rank perturbations are statistically observable, leading to a detailed characterization of the model’s singular geometry both locally and globally, and introduce a curvature‑corrected estimator for the RLCT.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Statistics ML.

Related stories

Hugging Face Trending Papers
Sep 2

LoRA-TSD: Tangent-Space Spectral Descent for LoRA via Muon-Style Updates

Low-rank adaptation (LoRA) is the standard way to fine-tune large models, yet when its two factors are trained independently, the update ignores the geometry of the low-rank weight change it induces. We introduce LoRA-TSD, an optimizer that treats every LoRA step as a tangent vector of the fixed-rank matrix manifold and takes the spectral-norm steepest-descent step of Muon inside that tangent space, mapping the result back to the factors through a retraction native to the LoRA parametrization.

llmsfine-tuningbenchmarks
More like this →
arXiv Machine Learning
Sep 3

LoRA-TSD: Tangent-Space Spectral Descent for LoRA via Muon-Style Updates

LoRA-TSD introduces a new optimizer for low‑rank adaptation (LoRA) that treats each update as a tangent vector on the fixed‑rank matrix manifold and applies a Muon‑style spectral‑norm steepest‑descent step within that tangent space. The method avoids costly full‑matrix operations and offers a retraction that is up to 2.8× cheaper than previous manifold approaches. The authors prove that their surrogate recovers LoRA‑Pro, identify the Riemannian gradient as the natural stationarity measure, and provide the first global convergence guarantees for both LoRA‑Pro and LoRA‑TSD, achieving superior performance across multiple benchmarks with Llama and Qwen models.

By Dmitrii Andriianov, Andrey Veprikov, Aleksandr Beznosikov
llmsfine-tuningbenchmarks
More like this →