The paper addresses the limitations of spectral dimension reduction for dynamical systems composed of weakly interacting components, where standard rank‑based methods either require exponentially many modes or omit entire components (a phenomenon termed linear masking). It proposes scoring the σ‑algebra generated by coordinates instead of individual modes, using a χ²‑divergence criterion that guarantees an embedding with twice the intrinsic dimension captures the full operator spectrum. Experiments on benchmark systems show that this algebraic approach recovers masked components and enables accurate prediction from few labels, outperforming traditional rank‑based and VAMP methods.
arXiv:2608. 15351v1 Announce Type: new Abstract: Nominal LoRA rank is a design parameter; calibrated spectral evidence is a separate inferential quantity.
By Mohammed Ahnouch, Lotfi Elaachak
arXiv:2609. 12259v1 Announce Type: new Abstract: Matrix-valued memories make rank the natural budget of a learned representation: the number of independent directions a state spans bounds what it can bind, compose, and track.
By Samuel Larson
LoRA-TSD introduces a new optimizer for low‑rank adaptation (LoRA) that treats each update as a tangent vector on the fixed‑rank matrix manifold and applies a Muon‑style spectral‑norm steepest‑descent step within that tangent space. The method avoids costly full‑matrix operations and offers a retraction that is up to 2.8× cheaper than previous manifold approaches. The authors prove that their surrogate recovers LoRA‑Pro, identify the Riemannian gradient as the natural stationarity measure, and provide the first global convergence guarantees for both LoRA‑Pro and LoRA‑TSD, achieving superior performance across multiple benchmarks with Llama and Qwen models.
By Dmitrii Andriianov, Andrey Veprikov, Aleksandr Beznosikov
Low-rank adaptation (LoRA) is the standard way to fine-tune large models, yet when its two factors are trained independently, the update ignores the geometry of the low-rank weight change it induces. We introduce LoRA-TSD, an optimizer that treats every LoRA step as a tangent vector of the fixed-rank matrix manifold and takes the spectral-norm steepest-descent step of Muon inside that tangent space, mapping the result back to the factors through a retraction native to the LoRA parametrization.
arXiv:2608. 11661v1 Announce Type: cross Abstract: A multiplicative dual-encoder network computes a real-valued output for a pair of inputs as the inner product of their separate encodings.
By Zijian Zhao, Sen Li