The paper proposes a new dimension‑reduction strategy for transfer‑operator models of dynamical systems that focuses on scoring the σ‑algebra generated by coordinates rather than the operator’s spectral span. By using a χ²‑divergence criterion between embedded present and future states, the method guarantees that twice the intrinsic system dimension suffices to capture the full operator spectrum, even for systems with weakly interacting components that would otherwise require exponentially many modes. Experiments on benchmark systems show that this algebraic approach recovers masked components missed by rank‑based methods and enables accurate prediction of those components from few labels.
By Mark Kozdoba, Shie Mannor
arXiv:2609.38901v1 Announce Type: new
Abstract: Representer explanations rank the training landmarks that most influence a self-supervised representation. At scale, this ranking rests on up to four s...
By Jayanta Mukherjee, Shourya Verma, Mengbo Wang, Jasorsi Ghosh, Ananth Grama
arXiv:2606. 01443v1 Announce Type: cross Abstract: A central difficulty in training Joint-Embedding Predictive Architectures (JEPAs) is preventing representation collapse.
By Triet M. Le
The paper investigates whether the rank of latent matrices in matrix‑chain‑of‑thought (Matrix‑CODI) models influences performance on reasoning tasks. Across multiple training regimes on ProsQA and GSM8K‑Aug, rank‑k projection ablations show flat accuracy curves, indicating that truncating the latent matrix to low rank does not hurt performance. Experiments with various readout architectures—bilinear, bilinear‑plus‑GELU, SVD‑augmented, and quadratic—confirm that rank‑indifference persists even for nonlinear readouts, and a linear probe on the latent matrix underperforms a raw pretrained hidden state.
By Samuel Larson (Pebble ML)
arXiv:2608.30374v1 Announce Type: cross
Abstract: We study null-space estimation from a noisy matrix. For a simple left null space, we first derive an exact compact expression for the error of the sm...
By Xin Li, Jonathan Cohen, Rami Puzis
The paper investigates whether the rank of matrix-valued latent representations in continuous chain‑of‑thought models influences task accuracy. Experiments on ProsQA and GSM8K‑Aug show that truncating the latent matrix to low rank has negligible effect (within 0.6 pp), and this flatness persists across various readout designs and even in a vanilla GPT‑2 baseline. The results suggest that rank is not a useful structural signal for these models’ reasoning paths.
The paper addresses the limitations of spectral dimension reduction for dynamical systems composed of weakly interacting components, where standard rank‑based methods either require exponentially many modes or omit entire components (a phenomenon termed linear masking). It proposes scoring the σ‑algebra generated by coordinates instead of individual modes, using a χ²‑divergence criterion that guarantees an embedding with twice the intrinsic dimension captures the full operator spectrum. Experiments on benchmark systems show that this algebraic approach recovers masked components and enables accurate prediction from few labels, outperforming traditional rank‑based and VAMP methods.
arXiv:2606. 13767v1 Announce Type: cross Abstract: Low-rank adaptation (LoRA) and its variants provide a memory- and compute-efficient alternative to full fine-tuning of pre-trained models.
By Elijah Cadenhead, Cristian McGee, Xin Li, El Houcine Bergou, Aritra Dutta
arXiv:2607. 16638v1 Announce Type: cross Abstract: Principal component regression (PCR) regularizes high-dimensional prediction by choosing a spectral cutoff, but rank selection cannot correct systematic inflation of the retained empirical eigenvalues.
By Peng Zhao
The paper introduces ISO-LoRA, an optimizer that improves rank utilization in Low‑Rank Adaptation (LoRA) by coupling factor updates through spectral descent on the induced tangent perturbation in weight space. Experiments on GPT‑2 adaptation show that standard optimizers like AdamW concentrate updates in a few singular directions, whereas ISO-LoRA distributes energy more evenly, leading to higher effective rank and better downstream performance across 0.1B‑7B models. The authors provide theoretical guarantees under a stylized spiked‑gradient model and demonstrate that ISO-LoRA consistently outperforms factor‑wise optimizers, especially at moderate‑to‑large LoRA ranks.
By Zihan Zhu, Zhehang Du, Xuyang Chen, Tim Tsz-Kit Lau, Jiayuan Wu, X. Y. Han, Qi Long, Weijie Su
arXiv:2606. 00428v1 Announce Type: cross Abstract: Low-rank adapters are usually compared by sweeping a small set of ranks, but the rank also fixes the resolution of the parameter budget.
By Xinjue Wang, Xiuheng Wang, Yejun Zhang, Sergiy A. Vorobyov, Esa Ollila, Zhi-Yong Wang
arXiv:2607. 27680v1 Announce Type: new Abstract: Low-Rank Adaptation (LoRA) has become the standard mechanism for fine-tuning large pretrained models, yet its statistical properties remain only partially understood.
By Arunan J