The paper introduces low‑rank orthogonalization, a technique that exploits the low‑rank nature of gradients in neural network training to perform matrix orthogonalization more efficiently. Building on this, the authors present low‑rank matrix‑signed gradient descent (MSGD) and a low‑rank variant of the Muon optimizer, showing through experiments that low‑rank Muon matches or surpasses vanilla Muon on GPT‑2 and LLaMA pretraining, especially for larger models. Theoretical analysis provides iteration‑complexity bounds for both low‑rank MSGD and low‑rank Muon under heavy‑tailed noise.
By Chuan He, Zhanwang Deng, Zhaosong Lu
arXiv:2109. 11057v2 Announce Type: replace-cross Abstract: Weighted low-rank matrix approximation (WLRMA) generalizes classical low-rank approximation and matrix completion by allowing arbitrary elementwise weights.
By Elena Tuzhilina, Trevor Hastie
arXiv:2602. 20376v3 Announce Type: replace-cross Abstract: We study the problem of maximizing a complex-valued quadratic form over the $K^{\text{th}}$ roots of unity.
By Ria Stevens, Fangshuo Liao, Barbara Su, Thanasis Hadjidimoulas, Jianqiang Li, Anastasios Kyrillidis
arXiv:2606. 02887v1 Announce Type: new Abstract: Symmetric nonnegative matrix factorization (Symmetric NMF) approximates a matrix as $WW^T$ with nonnegative rectangular factor $W$.
By Ryan Swart, Johannes Brust
arXiv:1312. 0925v4 Announce Type: replace Abstract: Alternating Minimization is a widely used and empirically successful heuristic for matrix completion and related low-rank optimization problems.
By Moritz Hardt
arXiv:2608. 13922v1 Announce Type: new Abstract: Detecting distributional changes in high dimension is difficult when neither the pre-change nor post-change density is parametrically specified.
By Guoqing Zhang, Zhaixin Chen
arXiv:2507.19290v2 Announce Type: replace-cross
Abstract: We study the problem of learning a structured approximation (low-rank, sparse, banded, etc.) to an unknown matrix $A$ given access to matrix-...
By Noah Amsel, Pratyush Avi, Tyler Chen, Feyza Duman Keles, Chinmay Hegde, Cameron Musco, Christopher Musco, David Persson
arXiv:2511. 11927v2 Announce Type: replace-cross Abstract: Principal Component Analysis (PCA) is a standard tool for extracting a low-rank signal from noisy observations.
By Urte Adomaityte, Gabriele Sicuro, Pierpaolo Vivo
arXiv:2512. 17473v3 Announce Type: replace-cross Abstract: We present an algorithm based on the alternating direction method of multipliers (ADMM) for solving nonlinear matrix decompositions (NMD).
By Atharva Awari, Nicolas Gillis, Arnaud Vandaele
LoRA-TSD introduces a new optimizer for low‑rank adaptation (LoRA) that treats each update as a tangent vector on the fixed‑rank matrix manifold and applies a Muon‑style spectral‑norm steepest‑descent step within that tangent space. The method avoids costly full‑matrix operations and offers a retraction that is up to 2.8× cheaper than previous manifold approaches. The authors prove that their surrogate recovers LoRA‑Pro, identify the Riemannian gradient as the natural stationarity measure, and provide the first global convergence guarantees for both LoRA‑Pro and LoRA‑TSD, achieving superior performance across multiple benchmarks with Llama and Qwen models.
By Dmitrii Andriianov, Andrey Veprikov, Aleksandr Beznosikov
Low-rank adaptation (LoRA) is the standard way to fine-tune large models, yet when its two factors are trained independently, the update ignores the geometry of the low-rank weight change it induces. We introduce LoRA-TSD, an optimizer that treats every LoRA step as a tangent vector of the fixed-rank matrix manifold and takes the spectral-norm steepest-descent step of Muon inside that tangent space, mapping the result back to the factors through a retraction native to the LoRA parametrization.
arXiv:2607. 18559v1 Announce Type: cross Abstract: Gaussian graphical model selection is usually studied under independent sampling, but in many applications the data arise as a single trajectory of a dependent stochastic process.
By Vignesh Tirukkonda, Gautam Dasarathy