Where Grokking Happens: Distributed Utility and Fourier Recoding Without a Module Switch
Read the original on arXiv AI →The paper investigates where the transition from memorization to generalization occurs within a Transformer. Using Transition Games—activation games with paired non-generalizing controls—it finds that distributed utility gain is driven by a block‑0 attention bias, with degree‑two modes explaining 67–92% of the contrast across replacement games. A detailed path study shows that block‑1 MLP mediates more of this effect than other downstream paths, contradicting the common "MLP memorizes, attention generalizes" hypothesis and suggesting that grokking is a spectral recoding of an existing distributed circuit rather than a module switch.
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.