Subspace Crystallization in Grokking: Rank Compression Tracks the Phase Transition
Read the original on arXiv AI →The paper investigates the relationship between rank compression of weight matrices and the phenomenon of grokking in neural networks. Using a mean‑field MLP and a one‑layer transformer trained on modular arithmetic, the authors find that rank compression occurs after the onset of grokking, not during memorization, and that the effective rank drops sharply around 900 steps following the grokking onset. They also observe a subspace crystallization effect, where the number of singular directions needed to preserve test accuracy shrinks dramatically during the transition, and demonstrate that artificially imposing low rank does not accelerate grokking, indicating that compression is a signature rather than a cause of the phenomenon.
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.