arXiv Machine Learning

The Gradient Does Not See Rank: Rank-Indifference in Matrix-CODI on ProsQA

The paper investigates whether the rank of latent matrices in matrix‑chain‑of‑thought (Matrix‑CODI) models influences performance on reasoning tasks. Across multiple training regimes on ProsQA and GSM8K‑Aug, rank‑k projection ablations show flat accuracy curves, indicating that truncating the latent matrix to low rank does not hurt performance. Experiments with various readout architectures—bilinear, bilinear‑plus‑GELU, SVD‑augmented, and quadratic—confirm that rank‑indifference persists even for nonlinear readouts, and a linear probe on the latent matrix underperforms a raw pretrained hidden state.

Hugging Face Trending Papers

Sep 2

The Gradient Does Not See Rank: Rank-Indifference in Matrix-CODI on ProsQA

The paper investigates whether the rank of matrix-valued latent representations in continuous chain‑of‑thought models influences task accuracy. Experiments on ProsQA and GSM8K‑Aug show that truncating the latent matrix to low rank has negligible effect (within 0.6 pp), and this flatness persists across various readout designs and even in a vanilla GPT‑2 baseline. The results suggest that rank is not a useful structural signal for these models’ reasoning paths.

llms fine-tuning benchmarks

arXiv Machine Learning

1d ago

When the Gradient Sees Rank: Provable Necessity, Causal Recruitment, and Composition in Trained Matrix Memories

arXiv:2609.17594v1 Announce Type: new Abstract: Can gradient-based training learn the rank needed to store and compose associations in a matrix memory? In our earlier study, we used a matrix-augmente...

By Samuel Larson

arXiv AI

Sep 10

Linear Algebra Foundations of Efficient Attention: A Phase Reversal in Rank Collapse Under SVD Compression

arXiv:2609.06341v1 Announce Type: cross Abstract: Linear algebra provides the framework of concepts (matrix rank, singular value decomposition (SVD), and eigendecomposition) that modern artificial in...

By Anjaneya Teja Sarma Kalvakolanu

llms

arXiv Machine Learning

Jun 16

Transformers Learn the Mestre-Nagao Heuristic

arXiv:2606. 15036v1 Announce Type: new Abstract: We train a two-layer transformer encoder to classify rational elliptic curves $E/\mathbb{Q}$ of conductor $\leq 10000$ as either rank 0 or rank 1 from the first 128 normalized Frobenius traces.

By Pranav Venkata Konda

llms rag safety

arXiv Machine Learning

4d ago

The Rank the Task Demands: A Causal Rank Law for Matrix Memories Trained on Group Composition

arXiv:2609. 12259v1 Announce Type: new Abstract: Matrix-valued memories make rank the natural budget of a learned representation: the number of independent directions a state spans bounds what it can bind, compose, and track.

By Samuel Larson

arXiv AI

Jun 24

Weight-Space Geometry of Offline Reasoning Training

arXiv:2606. 23740v1 Announce Type: cross Abstract: Offline reinforcement-learning losses (RFT, RIFT, DFT, Offline GRPO, DPO) are widely used to distill reasoning from large teachers into smaller students, and are typically compared on downstream accuracy alone.

By Aleksandr Nikolich, Igor Kiselev, Vladimir Platonov, Karina Romanova

reinforcement-learning fine-tuning

arXiv Machine Learning

Jul 8

No Subspace to Track: Non-Identifiability and Optimizer State in Low-Rank Training

arXiv:2607. 05872v1 Announce Type: new Abstract: Memory-efficient optimizers such as GaLore train large language models by projecting gradients onto a rank-r subspace recomputed every T steps, assuming this subspace is a slowly drifting object that can be tracked.

By Noel Thomas

llms computer-vision

arXiv Machine Learning

Aug 18

Spectral Rank Certification for Foundation Model Adapters

arXiv:2608. 15351v1 Announce Type: new Abstract: Nominal LoRA rank is a design parameter; calibrated spectral evidence is a separate inferential quantity.

By Mohammed Ahnouch, Lotfi Elaachak

fine-tuning

arXiv Machine Learning

Aug 6

The Loss Does Not See the Basis, but Adam Does

arXiv:2608. 05136v1 Announce Type: new Abstract: Gradient descent on a factored model $W = UV^\top$ is implicitly biased toward low-rank solutions, while Adam, starting from the same small initialization, is not.

By Devender Singh

llms safety

arXiv AI

Jun 30

DLR: Zero-Inference-Cost Latent Residuals for Low-Rank Pre-Training

arXiv:2606. 28932v1 Announce Type: cross Abstract: Large language models have driven recent progress in language and multimodal AI, yet pre-training them at scale is prohibitively expensive.

By Dong Wang, Wenwu Tang, Yun Cheng, Olga Saukh

llms fine-tuning multimodal benchmarks

arXiv Machine Learning

Sep 11

EGGROLL, Unrolled: Understanding and Improving Low-Rank Evolution Strategies at Scale

EGGROLL replaces dense Gaussian perturbations in evolution strategies with low‑rank Gaussian products, enabling practical optimization of large language models while maintaining exactness on quadratic objectives. The paper analyzes the mean update field, error bounds, and shows that rank‑one perturbations add only a small variance penalty compared to dense ES. A new leave‑one‑out estimator, LOO‑ROLL, further reduces estimator MSE and improves post‑training performance on transformer blocks and GSM8K accuracy.

By Ege C. Kaya, Abolfazl Hashemi

llms

arXiv Machine Learning

Jun 9

Shortcuts in the Tail: Debiasing via Post-Hoc Spectral Compression of Fine-Tuning Updates

arXiv:2606. 07596v1 Announce Type: new Abstract: Fine-tuning often introduces spurious correlations alongside task knowledge, causing systematic failures on underrepresented groups.

By Edward Sun, Dmitrii Troitskii

fine-tuning benchmarks

Related stories