The paper investigates how large language models (LLMs) perform multi‑hop reasoning and challenges the prevailing hop‑aligned circuit hypothesis, which posits that bridge entities are computed sequentially across layers. Through systematic analyses of real‑world multi‑hop queries, the authors discover a phenomenon called layer‑order inversion, where later‑hop answer entities become decodable earlier than bridge entities, and this effect grows with the number of hops. They propose a probabilistic recall‑and‑extract framework that models multi‑hop reasoning as broad probabilistic recall in shallow MLP layers followed by selective extraction in deeper attention layers, and validate this framework with probing analyses that reinterpret prior evidence, explain chain‑of‑thought gains, and diagnose multi‑hop failures.
By Xukai Liu, Ye Liu, Jipeng Zhang, Yanghai Zhang, Kai Zhang, Qi Liu
Large language models can solve complex multi‑hop tasks but often fail on simple two‑hop queries, even when each hop is individually correct. In a controlled symbolic setting, the authors find that models generalize reliably when the second hop follows the training distribution, but always fail when it deviates. Mechanistic analysis shows that successful generalization relies on consistent intermediate representations across contexts, whereas failures arise from a mismatch between lower‑layer representation construction and upper‑layer mapping to outputs. The study proposes a recurrent‑style training strategy that improves out‑of‑distribution two‑hop generalization.
By Zili Zhang, Yilin Wang, Heng Wang, Herun Wan, Minnan Luo
arXiv:2607. 00341v1 Announce Type: cross Abstract: Large language models achieve strong performance on many reasoning tasks when allowed to externalize intermediate steps as Chain-of-Thought (CoT).
By Hengyu Fu, Tianyu Guo, Zixuan Wang, Hanlin Zhu, Jason D. Lee, Jiantao Jiao, Stuart Russell, Song Mei
arXiv:2604. 07822v2 Announce Type: replace-cross Abstract: We study implicit reasoning, i.
By Harsh Kohli, Srinivasan Parthasarathy, Huan Sun, Yuekun Yao
arXiv:2607. 20500v1 Announce Type: new Abstract: Large Language Models (LLMs) perform strongly on well-specified reasoning tasks with a feasible answer.
By Sizhe Tang, Guangyu Jiang, Yu Li, Rongqian Chen, Ioannis G. Kevrekidis, Tian Lan
Large Language Models fail at implicit multi-hop reasoning: a model answers "When was $X$ born? " and "Who is $Y$'s closest friend?
Large Language Models struggle with implicit multi‑hop reasoning, correctly answering individual facts but failing to combine them in a single pass. In a controlled setting, the authors show that this failure persists even with high 1‑hop accuracy, indicating it is due to pretraining exposure rather than missing knowledge. They test nine data‑centric augmentation formats and find that only individuals seen in compositional contexts during pretraining enable transfer to unseen questions, proving exposure to such contexts is necessary for implicit multi‑hop reasoning.
By Yannis Karmim, Luis Marti, Djam\'e Seddah, Valentin Barri\`ere
arXiv:2601.07794v2 Announce Type: replace-cross
Abstract: Large language models (LLMs) are increasingly evaluated on their ability to perform multi-hop reasoning, i.e., to combine multiple pieces of...
By Tianda Sun, Dimitar Kazakov
arXiv:2505. 20161v2 Announce Type: replace-cross Abstract: Effective generalization in language models depends critically on the diversity of their training data.
By Jaehun Jung, Seungju Han, Ximing Lu, Skyler Hallinan, David Acuna, Shrimai Prabhumoye, Mostafa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Yejin Choi
arXiv:2604. 11912v2 Announce Type: replace-cross Abstract: While next-token prediction (NTP) has been the standard objective for training language models, it often struggles to capture global structure in reasoning tasks.
By Jianhao Huang, Zhanpeng Zhou, Renqiu Xia, Baharan Mirzasoleiman, Weijie Su, Wei Huang
arXiv:2606. 05181v1 Announce Type: cross Abstract: Natural Language Inference (NLI) is a fundamental task in natural language understanding that requires determining the logical relationship between a premise and a hypothesis.
By Chunling Xi, Di Liang
We present a theoretical framework to explain the emergence of inductive reasoning abilities in Transformer language models. While previous works on Transformer learning dynamics have so far been mostly tied to specific tasks, we study a generalized class of inductive tasks that unifies several synthetic tasks known in the literature, including in-context n-grams and multi-hop reasoning.