← Back to all news
arXiv Machine Learning By Samuel Larson (Pebble ML)

The Gradient Does Not See Rank: Rank-Indifference in Matrix-CODI on ProsQA

Read the original on arXiv Machine Learning →

The paper investigates whether the rank of latent matrices in matrix‑chain‑of‑thought (Matrix‑CODI) models influences performance on reasoning tasks. Across multiple training regimes on ProsQA and GSM8K‑Aug, rank‑k projection ablations show flat accuracy curves, indicating that truncating the latent matrix to low rank does not hurt performance. Experiments with various readout architectures—bilinear, bilinear‑plus‑GELU, SVD‑augmented, and quadratic—confirm that rank‑indifference persists even for nonlinear readouts, and a linear probe on the latent matrix underperforms a raw pretrained hidden state.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

Hugging Face Trending Papers
Sep 2

The Gradient Does Not See Rank: Rank-Indifference in Matrix-CODI on ProsQA

The paper investigates whether the rank of matrix-valued latent representations in continuous chain‑of‑thought models influences task accuracy. Experiments on ProsQA and GSM8K‑Aug show that truncating the latent matrix to low rank has negligible effect (within 0.6 pp), and this flatness persists across various readout designs and even in a vanilla GPT‑2 baseline. The results suggest that rank is not a useful structural signal for these models’ reasoning paths.

llmsfine-tuningbenchmarks
More like this →
arXiv Machine Learning
Sep 17

When the Gradient Sees Rank: Provable Necessity, Causal Recruitment, and Composition in Trained Matrix Memories

The paper investigates whether gradient-based training can learn the necessary rank to store and compose associations in a matrix memory. By training on $K$ key‑value bindings that require rank at least $K$ for exact linear recovery, the authors show that the learned effective rank increases with $K$ and that recovery sharply transitions near $k=K$ when rank is capped. Experiments demonstrate high recovery rates (cosine similarity >0.9) across multiple seeds and show that the trained operator approximates the ideal cycle on the entity subspace, though performance declines with larger matrix dimensions when encoder width is fixed.

By Samuel Larson
More like this →