← Back to all news
arXiv Machine Learning

The Gradient Does Not See Rank: Rank-Indifference in Matrix-CODI on ProsQA

The paper investigates whether the rank of latent matrices in matrix‑chain‑of‑thought (Matrix‑CODI) models influences performance on reasoning tasks. Across multiple training regimes on ProsQA and GSM8K‑Aug, rank‑k projection ablations show flat accuracy curves, indicating that truncating the latent matrix to low rank does not hurt performance. Experiments with various readout architectures—bilinear, bilinear‑plus‑GELU, SVD‑augmented, and quadratic—confirm that rank‑indifference persists even for nonlinear readouts, and a linear probe on the latent matrix underperforms a raw pretrained hidden state.

Related stories

Hugging Face Trending Papers
Sep 2

The Gradient Does Not See Rank: Rank-Indifference in Matrix-CODI on ProsQA

The paper investigates whether the rank of matrix-valued latent representations in continuous chain‑of‑thought models influences task accuracy. Experiments on ProsQA and GSM8K‑Aug show that truncating the latent matrix to low rank has negligible effect (within 0.6 pp), and this flatness persists across various readout designs and even in a vanilla GPT‑2 baseline. The results suggest that rank is not a useful structural signal for these models’ reasoning paths.

llmsfine-tuningbenchmarks
More like this →
arXiv Machine Learning
Sep 17

When the Gradient Sees Rank: Provable Necessity, Causal Recruitment, and Composition in Trained Matrix Memories

The paper investigates whether gradient-based training can learn the necessary rank to store and compose associations in a matrix memory. By training on $K$ key‑value bindings that require rank at least $K$ for exact linear recovery, the authors show that the learned effective rank increases with $K$ and that recovery sharply transitions near $k=K$ when rank is capped. Experiments demonstrate high recovery rates (cosine similarity >0.9) across multiple seeds and show that the trained operator approximates the ideal cycle on the entity subspace, though performance declines with larger matrix dimensions when encoder width is fixed.

By Samuel Larson
More like this →
arXiv Machine Learning
5d ago

SAKI-RoPE: Score-Aware Low-Rank Key Indexing with RoPE-Aware Covariance and Random-Matrix Noise Analysis

The paper introduces SAKI-RoPE, a training‑free low‑rank key‑indexing method that optimally preserves attention scores by solving a covariance‑weighted rank‑r problem using query and key statistics. It outperforms key‑PCA across multiple large language models and context lengths, achieving higher recall with significantly lower rank. The authors also analyze random‑matrix noise to justify the calibration token budget and demonstrate that clipping the Marchenko‑Pastur bulk improves performance when calibration data is limited.

By Lin Zhang
llms
More like this →
arXiv Computation and Language
Oct 1

Learning Functional Subspaces for Neural Network Compression

arXiv:2609.40127v1 Announce Type: cross Abstract: Modern transformers pair impressive capabilities with substantial memory and compute demands. Low-rank weight factorization reduces both while keepin...

By Massimo Bini, Anders Christensen, Stephan Alaniz, Judah Goldfeder, Ole Winther, Yann LeCun, Ravid Shwartz-Ziv, Zeynep Akata
llmsefficiency
More like this →