arXiv Computation and Language
The Emergence and Sparsification of a Syntactic Circuit
Read the original on arXiv Computation and Language →The Flow has not summarised this story yet — read it at arXiv Computation and Language.
Related stories
arXiv Machine Learning
Jun 2
A Pre-Training Analogue of Grokking in Language Models: Tracing Delayed Grammatical Generalization
arXiv:2606. 00230v1 Announce Type: new Abstract: Grokking, the phenomenon in which neural networks generalize long after fitting their training data, has been studied in supervised settings on many epochs.
arXiv Computation and Language
Oct 21
Are Linguistic and Arithmetic Circuits Dissociated in LLMs?
The paper investigates whether linguistic and arithmetic tasks in large language models (LLMs) rely on shared neural circuits. Using mechanistic interpretability techniques on five LLMs, the authors find that circuit similarity aligns primarily with task domain rather than input/output vocabularies or syntactic structure. This suggests that LLMs may employ largely separate neural circuits for language and arithmetic, indicating a modular organization of abilities.
arXiv Computation and Language
Oct 22
Word-Class and Construction-Like Structure Emerges in Neural Successor Representations Trained on Natural Language
arXiv:2605.24585v3 Announce Type: replace Abstract: Neural language models are typically trained on next-token prediction, although linguistic structure spans multiple temporal scales. Successor repr...
arXiv Machine Learning
Oct 15
Tracing Morphology, Semantics, and Divergence in Language Acquisition
arXiv:2610.19041v1 Announce Type: new Abstract: Understanding the dynamics of language acquisition in Large Language Models (LLMs) is crucial for uncovering how statistical representations evolve int...
arXiv Computation and Language
Sep 15
Where Animacy Lives in Large Language Models: Tracing the Circuits of the Animacy Concept
arXiv:2607.20995v2 Announce Type: replace Abstract: Distinguishing animate from inanimate concepts in written language requires more than shallow text processing, as it involves recognizing complex s...
arXiv AI
Oct 16
Learning to Read Out: Unembedding Dynamics in Language Model Pretraining
The paper investigates when a language model actually learns task information versus when its readout (unembedding matrix) learns to express that information. Using a new parameter‑trajectory cross‑coding method, the authors find that hidden states can encode task data early in training, but the readout reorganizes only in a narrow window (0.4–1.1% of Pythia’s pretraining). By varying the readout learning rate, they show that faster readout learning reduces the gap between hidden‑state encoding and output expression without affecting final performance.