← Back to all news
Hugging Face Trending Papers

The Gradient Does Not See Rank: Rank-Indifference in Matrix-CODI on ProsQA

The paper investigates whether the rank of matrix-valued latent representations in continuous chain‑of‑thought models influences task accuracy. Experiments on ProsQA and GSM8K‑Aug show that truncating the latent matrix to low rank has negligible effect (within 0.6 pp), and this flatness persists across various readout designs and even in a vanilla GPT‑2 baseline. The results suggest that rank is not a useful structural signal for these models’ reasoning paths.

Related stories

arXiv Machine Learning
Sep 4

The Gradient Does Not See Rank: Rank-Indifference in Matrix-CODI on ProsQA

The paper investigates whether the rank of latent matrices in matrix‑chain‑of‑thought (Matrix‑CODI) models influences performance on reasoning tasks. Across multiple training regimes on ProsQA and GSM8K‑Aug, rank‑k projection ablations show flat accuracy curves, indicating that truncating the latent matrix to low rank does not hurt performance. Experiments with various readout architectures—bilinear, bilinear‑plus‑GELU, SVD‑augmented, and quadratic—confirm that rank‑indifference persists even for nonlinear readouts, and a linear probe on the latent matrix underperforms a raw pretrained hidden state.

By Samuel Larson (Pebble ML)
llmsfine-tuningbenchmarks
More like this →
arXiv Machine Learning
Sep 17

When the Gradient Sees Rank: Provable Necessity, Causal Recruitment, and Composition in Trained Matrix Memories

The paper investigates whether gradient-based training can learn the necessary rank to store and compose associations in a matrix memory. By training on $K$ key‑value bindings that require rank at least $K$ for exact linear recovery, the authors show that the learned effective rank increases with $K$ and that recovery sharply transitions near $k=K$ when rank is capped. Experiments demonstrate high recovery rates (cosine similarity >0.9) across multiple seeds and show that the trained operator approximates the ideal cycle on the entity subspace, though performance declines with larger matrix dimensions when encoder width is fixed.

By Samuel Larson
More like this →
arXiv Computation and Language
Sep 2

Behaviorally Effective LoRA Writes Are Sparse and Structured

The paper investigates which parts of a low‑rank adaptation (LoRA) update actually drive behavior in language models. It shows that behaviorally effective LoRA writes are sparse, structured, and highly concentrated, with only a few module‑wise components carrying most of the signal. Using a learned‑basis continuation recipe, the authors demonstrate that these sparse, structured writes can be recovered with minimal error and that they remain robust across different training continuations and evaluation tasks.

By Haruto Sato, Yuki Tanaka, Ren Nakamura, Aoi Kobayashi, Mei Ito
fine-tuning
More like this →
Hugging Face Trending Papers
Sep 2

LoRA-TSD: Tangent-Space Spectral Descent for LoRA via Muon-Style Updates

Low-rank adaptation (LoRA) is the standard way to fine-tune large models, yet when its two factors are trained independently, the update ignores the geometry of the low-rank weight change it induces. We introduce LoRA-TSD, an optimizer that treats every LoRA step as a tangent vector of the fixed-rank matrix manifold and takes the spectral-norm steepest-descent step of Muon inside that tangent space, mapping the result back to the factors through a retraction native to the LoRA parametrization.

llmsfine-tuningbenchmarks
More like this →