We present a theoretical framework to explain the emergence of inductive reasoning abilities in Transformer language models. While previous works on Transformer learning dynamics have so far been mostly tied to specific tasks, we study a generalized class of inductive tasks that unifies several synthetic tasks known in the literature, including in-context n-grams and multi-hop reasoning.
arXiv:2510. 25013v2 Announce Type: replace-cross Abstract: Mechanistic interpretability aims to reverse-engineer large language models (LLMs) into human-understandable computational circuits.
By Rabin Adhikari
The paper investigates how transformer models learn latent structure by training a small decoder-only transformer on three variants of the Alchemy benchmark. It finds that the model acquires different components of latent structure in discrete stages, with a notable asymmetry: it robustly composes fundamental transitions but struggles to decompose complex examples into atomic transitions. Layer‑specific causal interventions reveal plasticity windows where freezing layers delays or prevents stage completion, offering a detailed view of capability evolution during training.
By Rohan Saha, Farzane Aminmansour, Alona Fyshe
Large language models can solve complex multi‑hop tasks but often fail on simple two‑hop queries, even when each hop is individually correct. In a controlled symbolic setting, the authors find that models generalize reliably when the second hop follows the training distribution, but always fail when it deviates. Mechanistic analysis shows that successful generalization relies on consistent intermediate representations across contexts, whereas failures arise from a mismatch between lower‑layer representation construction and upper‑layer mapping to outputs. The study proposes a recurrent‑style training strategy that improves out‑of‑distribution two‑hop generalization.
By Zili Zhang, Yilin Wang, Heng Wang, Herun Wan, Minnan Luo
arXiv:2601. 15158v4 Announce Type: replace-cross Abstract: Transformers trained via Reinforcement Learning (RL) with outcome-based supervision can spontaneously develop the ability to generate intermediate reasoning steps (Chain-of-Thought).
By Yuval Ran-Milo, Yotam Alexander, Shahar Mendel, Nadav Cohen
arXiv:2604. 07822v2 Announce Type: replace-cross Abstract: We study implicit reasoning, i.
By Harsh Kohli, Srinivasan Parthasarathy, Huan Sun, Yuekun Yao
arXiv:2607. 15178v1 Announce Type: cross Abstract: Transformer reasoning is limited by autoregressive decoding, which repeat edly compresses rich hidden computation through token space and makes it difficult for intermediate reasoning states to persist across time.
By Ziyang Cai, Xingyu Zhu, Yihe Dong, Yinghui He, Sanjeev Arora
arXiv:2609.08981v1 Announce Type: cross
Abstract: A growing body of work establishes that large language models are not mere statistical memorizers, but are capable of in-context learning: performing...
By Arman Adibi, Alireza Jafari, Mohammad Ghavamzadeh, Hadi Daneshmand
arXiv:2602. 22600v2 Announce Type: replace-cross Abstract: Training selects for behavior, not circuitry: many weight configurations can implement the same function.
By Joshua S. Schiffman
arXiv:2607. 07316v1 Announce Type: new Abstract: This article offers a comprehensive overview of mechanistic interpretability, an emerging field that seeks to reverse-engineer the internal algorithms of modern neural networks.
By Pranav Sawant, Jakub Krej\v{c}\'i
Transformers can learn broad families of tasks during pretraining and adapt to unseen tasks from a short prompt, but a rigorous understanding of this capability is limited. This paper studies how shared cross‑task structure influences the sample complexity of in‑context learning (ICL) by characterizing task‑space complexity through covering numbers, yielding a set of anchor functions that localize unseen tasks and predict responses. The authors construct a Transformer with Softmax attention to approximate this procedure and derive an error bound that separates the effects of pretraining tasks and prompt length, showing that once enough tasks are available the dependence on prompt length becomes dimension‑free.
By Zhongjie Shi, Rongjie Lai, Alexander Cloninger, Wenjing Liao
arXiv:2607. 19405v1 Announce Type: new Abstract: The CoTFormer architecture formalizes Chain-of-Thought as a form of recurrent latent computation, preserving intermediate states as attendable representations to mimic explicit reasoning traces.
By Aras Kavuncu, Bryan Vullo, Alberto Berni