← Back to all news
arXiv Computation and Language By Zili Zhang, Yilin Wang, Heng Wang, Herun Wan, Minnan Luo

Why Knowing Both Hops Is Not Enough: Understanding Two-Hop Generalization in Language Models

Read the original on arXiv Computation and Language →

Large language models can solve complex multi‑hop tasks but often fail on simple two‑hop queries, even when each hop is individually correct. In a controlled symbolic setting, the authors find that models generalize reliably when the second hop follows the training distribution, but always fail when it deviates. Mechanistic analysis shows that successful generalization relies on consistent intermediate representations across contexts, whereas failures arise from a mismatch between lower‑layer representation construction and upper‑layer mapping to outputs. The study proposes a recurrent‑style training strategy that improves out‑of‑distribution two‑hop generalization.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv Computation and Language
Aug 27

Layer-Order Inversion: Rethinking Latent Multi-Hop Reasoning in Large Language Models

The paper investigates how large language models (LLMs) perform multi‑hop reasoning and challenges the prevailing hop‑aligned circuit hypothesis, which posits that bridge entities are computed sequentially across layers. Through systematic analyses of real‑world multi‑hop queries, the authors discover a phenomenon called layer‑order inversion, where later‑hop answer entities become decodable earlier than bridge entities, and this effect grows with the number of hops. They propose a probabilistic recall‑and‑extract framework that models multi‑hop reasoning as broad probabilistic recall in shallow MLP layers followed by selective extraction in deeper attention layers, and validate this framework with probing analyses that reinterpret prior evidence, explain chain‑of‑thought gains, and diagnose multi‑hop failures.

By Xukai Liu, Ye Liu, Jipeng Zhang, Yanghai Zhang, Kai Zhang, Qi Liu
llms
More like this →
arXiv Computation and Language
Sep 17

Multi-Hop Knowledge Composition is Bound by Pretraining Exposure

Large Language Models struggle with implicit multi‑hop reasoning, correctly answering individual facts but failing to combine them in a single pass. In a controlled setting, the authors show that this failure persists even with high 1‑hop accuracy, indicating it is due to pretraining exposure rather than missing knowledge. They test nine data‑centric augmentation formats and find that only individuals seen in compositional contexts during pretraining enable transfer to unseen questions, proving exposure to such contexts is necessary for implicit multi‑hop reasoning.

By Yannis Karmim, Luis Marti, Djam\'e Seddah, Valentin Barri\`ere
llms
More like this →