arXiv Computation and Language

Detecting Differences Is Not Understanding Structure: Large Language Models Fail at Graph Isomorphism

arXiv Computation and Language
Oct 21

Diagnostic Analysis for Large Language Model Reasoning

The paper introduces a two-stage diagnostic evaluation to assess the structural reliability of reasoning graphs used in large language models for tasks like multi-hop question answering. Stage 1 examines how graph reliability depends on topology, prompt design, and model family, while Stage 2 evaluates the impact of graph errors on uncertainty estimation. Findings show that graph-based computation is vulnerable to upstream errors, offers limited benefit as prompt context alone, but improves confidence-based answer discrimination and enables node-level diagnostics.

By Xinyu Liu, Dongqi Fu, Jun Wu
arXiv AI
Aug 14

Unified Multi-Dimensional Benchmark for Complex Graph Reasoning in Large Language Models

arXiv:2608. 12391v1 Announce Type: cross Abstract: Graph reasoning provides a promising testbed for evaluating the reasoning ability of large language models (LLMs), as graph instances can be programmatically generated, structurally controlled, and naturally scaled to long-input settings.

By Fali Wang, Ali Al-Lawati, Iliyas Bektas, Jinxuan Fang, Alek Melenski, Tianxiang Zhao, Yao Ma, Suhang Wang
arXiv AI
Sep 7

GUT: Quantifying and Optimizing the Reasoning Uncertainty of LLMs via Graph Complexity

The paper introduces GUT, a method that uses directed acyclic graphs to represent all possible reasoning branches of Large Language Models (LLMs). It comprises two modules: GUT-Q, which quantifies reasoning uncertainty by approximating graph complexity, and GUT-O, which reduces uncertainty through reinforcement learning that rewards lower uncertainty. Experiments on four LLMs across five datasets demonstrate GUT’s effectiveness in measuring and mitigating reasoning uncertainty.

By Shuang Liang, Xin-Yu Hu, Xiang-Jun Ou, Shao-Qun Zhang