arXiv AI October 31, 2026 By Davide Venuto, Leonardo Venuto

Implicit Reasoning in Transformers: Bridging the Compositionality Gap in GPT-2-like Models

The paper examines the compositionality gap in GPT‑2‑like transformers, where models can combine known facts in‑distribution but fail to generalize to out‑of‑distribution knowledge. By aligning the training distribution so that both ID and OOD atomic facts appear in the same positional format as the second‑hop retrieval step of a known compositional circuit, the authors show that the model can transfer its learned compositional behavior to unseen OOD facts. This demonstrates that the gap is not an inherent architectural limitation but can be addressed through targeted training distribution design.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

llms

Hugging Face Trending Papers

4d ago

Implicit Reasoning in Transformers: Bridging the Compositionality Gap in GPT-2-like Models

Compositional generalization remains a persistent limitation of transformer language models: even when a model learns to combine known facts correctly in-distribution (ID), the same compositional mech...

llms

arXiv Computation and Language

Aug 28

Why Knowing Both Hops Is Not Enough: Understanding Two-Hop Generalization in Language Models

Large language models can solve complex multi‑hop tasks but often fail on simple two‑hop queries, even when each hop is individually correct. In a controlled symbolic setting, the authors find that models generalize reliably when the second hop follows the training distribution, but always fail when it deviates. Mechanistic analysis shows that successful generalization relies on consistent intermediate representations across contexts, whereas failures arise from a mismatch between lower‑layer representation construction and upper‑layer mapping to outputs. The study proposes a recurrent‑style training strategy that improves out‑of‑distribution two‑hop generalization.

By Zili Zhang, Yilin Wang, Heng Wang, Herun Wan, Minnan Luo

llms

arXiv AI

Jul 14

Unveiling the Mechanisms of Multi-Hop Reasoning in Transformers via Identity Bridge

arXiv:2509. 24653v2 Announce Type: replace-cross Abstract: Large Language Models (LLMs) excel at multi-hop reasoning in distribution, yet fail on unseen compositions, a phenomenon known as the curse of two-hop reasoning.

By Pengxiao Lin, Zheng-An Chen, Zhi-Qin John Xu

llms fine-tuning

arXiv AI

Aug 12

Loop, Think, & Generalize: Implicit Reasoning in Recurrent-Depth Transformers

arXiv:2604. 07822v2 Announce Type: replace-cross Abstract: We study implicit reasoning, i.

By Harsh Kohli, Srinivasan Parthasarathy, Huan Sun, Yuekun Yao

llms

arXiv AI

Jul 14

Invariant Learning Dynamics of Transformers in Inductive Reasoning Tasks

arXiv:2607. 11875v1 Announce Type: cross Abstract: We present a theoretical framework to explain the emergence of inductive reasoning abilities in Transformer language models.

By Tiberiu Musat, Tiago Pimentel, Nicholas Zucchet, Thomas Hofmann

llms

arXiv AI

Sep 1

Learning Composable Chains-of-Thought

arXiv:2505.22635v2 Announce Type: replace-cross Abstract: A common approach for teaching large language models (LLMs) to reason is to train on chain-of-thought (CoT) traces of in-distribution reasoni...

By Fangcong Yin, Zeyu Leo Liu, Liu Leqi, Xi Ye, Greg Durrett

llms fine-tuning

One email a morning, machine-written

Related stories

Implicit Reasoning in Transformers: Bridging the Compositionality Gap in GPT-2-like Models

Why Knowing Both Hops Is Not Enough: Understanding Two-Hop Generalization in Language Models

Unveiling the Mechanisms of Multi-Hop Reasoning in Transformers via Identity Bridge

Loop, Think, & Generalize: Implicit Reasoning in Recurrent-Depth Transformers

Invariant Learning Dynamics of Transformers in Inductive Reasoning Tasks

Learning Composable Chains-of-Thought