← Back to all news
arXiv Machine Learning

Score the Algebra, Not the Span: Dimension Reduction for Transfer Operator Models of Dynamical Systems

The paper proposes a new dimension‑reduction strategy for transfer‑operator models of dynamical systems that focuses on scoring the σ‑algebra generated by coordinates rather than the operator’s spectral span. By using a χ²‑divergence criterion between embedded present and future states, the method guarantees that twice the intrinsic system dimension suffices to capture the full operator spectrum, even for systems with weakly interacting components that would otherwise require exponentially many modes. Experiments on benchmark systems show that this algebraic approach recovers masked components missed by rank‑based methods and enables accurate prediction of those components from few labels.

Related stories

Hugging Face Trending Papers
Aug 19

Score the Algebra, Not the Span: Dimension Reduction for Transfer Operator Models of Dynamical Systems

The paper addresses the limitations of spectral dimension reduction for dynamical systems composed of weakly interacting components, where standard rank‑based methods either require exponentially many modes or omit entire components (a phenomenon termed linear masking). It proposes scoring the σ‑algebra generated by coordinates instead of individual modes, using a χ²‑divergence criterion that guarantees an embedding with twice the intrinsic dimension captures the full operator spectrum. Experiments on benchmark systems show that this algebraic approach recovers masked components and enables accurate prediction from few labels, outperforming traditional rank‑based and VAMP methods.

ragbenchmarks
More like this →
arXiv Machine Learning
Oct 12

A Categorical Atlas of Parameter-Efficient Fine-Tuning: Cones, Orbits, and Gauges

The paper presents a comprehensive framework for comparing parameter‑efficient fine‑tuning (PEFT) methods by treating each as a smooth map from a small parameter space into weight space, with a pretrained checkpoint as the base point. It identifies three key invariants—image near the base point, fibre dynamics, and base change—that explain expressivity, training dynamics, and initialization behavior. The authors prove numerous results about common PEFT techniques (e.g., LoRA, Householder adaptation) and classify 382 methods from 138 papers using seven coordinate descriptors.

By Xiaoyu Li, Zhizhou Sha, Chiwun Yang, Dai Shi
fine-tuningbenchmarks
More like this →
arXiv Machine Learning
Sep 3

LoRA-TSD: Tangent-Space Spectral Descent for LoRA via Muon-Style Updates

LoRA-TSD introduces a new optimizer for low‑rank adaptation (LoRA) that treats each update as a tangent vector on the fixed‑rank matrix manifold and applies a Muon‑style spectral‑norm steepest‑descent step within that tangent space. The method avoids costly full‑matrix operations and offers a retraction that is up to 2.8× cheaper than previous manifold approaches. The authors prove that their surrogate recovers LoRA‑Pro, identify the Riemannian gradient as the natural stationarity measure, and provide the first global convergence guarantees for both LoRA‑Pro and LoRA‑TSD, achieving superior performance across multiple benchmarks with Llama and Qwen models.

By Dmitrii Andriianov, Andrey Veprikov, Aleksandr Beznosikov
llmsfine-tuningbenchmarks
More like this →
Hugging Face Trending Papers
Sep 2

LoRA-TSD: Tangent-Space Spectral Descent for LoRA via Muon-Style Updates

Low-rank adaptation (LoRA) is the standard way to fine-tune large models, yet when its two factors are trained independently, the update ignores the geometry of the low-rank weight change it induces. We introduce LoRA-TSD, an optimizer that treats every LoRA step as a tangent vector of the fixed-rank matrix manifold and takes the spectral-norm steepest-descent step of Muon inside that tangent space, mapping the result back to the factors through a retraction native to the LoRA parametrization.

llmsfine-tuningbenchmarks
More like this →