The Flow has not summarised this story yet — read it at Hugging Face Trending Papers.
The paper examines the compositionality gap in GPT‑2‑like transformers, where models can combine known facts in‑distribution but fail to generalize to out‑of‑distribution knowledge. By aligning the training distribution so that both ID and OOD atomic facts appear in the same positional format as the second‑hop retrieval step of a known compositional circuit, the authors show that the model can transfer its learned compositional behavior to unseen OOD facts. This demonstrates that the gap is not an inherent architectural limitation but can be addressed through targeted training distribution design.
Large language models can solve complex multi‑hop tasks but often fail on simple two‑hop queries, even when each hop is individually correct. In a controlled symbolic setting, the authors find that models generalize reliably when the second hop follows the training distribution, but always fail when it deviates. Mechanistic analysis shows that successful generalization relies on consistent intermediate representations across contexts, whereas failures arise from a mismatch between lower‑layer representation construction and upper‑layer mapping to outputs. The study proposes a recurrent‑style training strategy that improves out‑of‑distribution two‑hop generalization.
arXiv:2505.22635v2 Announce Type: replace-cross Abstract: A common approach for teaching large language models (LLMs) to reason is to train on chain-of-thought (CoT) traces of in-distribution reasoni...
arXiv:2509. 24653v2 Announce Type: replace-cross Abstract: Large Language Models (LLMs) excel at multi-hop reasoning in distribution, yet fail on unseen compositions, a phenomenon known as the curse of two-hop reasoning.
arXiv:2604. 07822v2 Announce Type: replace-cross Abstract: We study implicit reasoning, i.
arXiv:2607. 11875v1 Announce Type: cross Abstract: We present a theoretical framework to explain the emergence of inductive reasoning abilities in Transformer language models.