Understanding the Staged Dynamics of Transformers in Learning Latent Structure
Read the original on arXiv Machine Learning →The paper investigates how transformer models learn latent structure by training a small decoder-only transformer on three variants of the Alchemy benchmark. It finds that the model acquires different components of latent structure in discrete stages, with a notable asymmetry: it robustly composes fundamental transitions but struggles to decompose complex examples into atomic transitions. Layer‑specific causal interventions reveal plasticity windows where freezing layers delays or prevents stage completion, offering a detailed view of capability evolution during training.
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.