arXiv Computation and Language October 21, 2026 By Xinyu Liu, Dongqi Fu, Jun Wu

Diagnostic Analysis for Large Language Model Reasoning

Read the original on arXiv Computation and Language →

The paper introduces a two-stage diagnostic evaluation to assess the structural reliability of reasoning graphs used in large language models for tasks like multi-hop question answering. Stage 1 examines how graph reliability depends on topology, prompt design, and model family, while Stage 2 evaluates the impact of graph errors on uncertainty estimation. Findings show that graph-based computation is vulnerable to upstream errors, offers limited benefit as prompt context alone, but improves confidence-based answer discrimination and enables node-level diagnostics.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

llms

arXiv AI

Jun 9

Diagnosing Multi-step Reasoning Failures in Black-box LLMs via Stepwise Confidence Attribution

arXiv:2605. 19228v2 Announce Type: replace-cross Abstract: Large Language Models have achieved strong performance on reasoning tasks with objective answers by generating step-by-step solutions, but diagnosing where a multi-step reasoning trace might fail remains difficult.

By Xiaoou Liu, Tiejin Chen, Dengjia Zhang, Yaqing Wang, Lu Cheng, Hua Wei

llms nlp

arXiv AI

Jun 3

Reasoning Structure of Large Language Models

arXiv:2606. 03883v1 Announce Type: new Abstract: Large reasoning models (LRMs) are often evaluated using metrics such as final-answer accuracy or token count.

By Fr\'ed\'eric Berdoz, Luca A. Lanzend\"orfer, Fabian Farestam, Roger Wattenhofer

llms benchmarks

Hugging Face Trending Papers

Aug 31

Graph Evidence Is Not Enough: Diagnosing Native Decoder Use in Graph-Augmented LLMs

Graph-augmented large language models often assume that graph evidence produced by external computation and placed in the input can be used by the native decoder. We test this assumption with HopQA, a...

llms safety

arXiv AI

Sep 7

GUT: Quantifying and Optimizing the Reasoning Uncertainty of LLMs via Graph Complexity

The paper introduces GUT, a method that uses directed acyclic graphs to represent all possible reasoning branches of Large Language Models (LLMs). It comprises two modules: GUT-Q, which quantifies reasoning uncertainty by approximating graph complexity, and GUT-O, which reduces uncertainty through reinforcement learning that rewards lower uncertainty. Experiments on four LLMs across five datasets demonstrate GUT’s effectiveness in measuring and mitigating reasoning uncertainty.

By Shuang Liang, Xin-Yu Hu, Xiang-Jun Ou, Shao-Qun Zhang

llms reinforcement-learning

arXiv AI

Aug 26

Omanic: Towards Step-wise Evaluation of Multi-hop Reasoning in Large Language Models

arXiv:2603.16654v3 Announce Type: replace-cross Abstract: Evaluating the reasoning abilities of large language models (LLMs) solely from final answers can obscure failures in intermediate steps, espe...

By Xiaojie Gu, Sherry T. Tong, Aosong Feng, Sophia Simeng Han, Jinghui Lu, Yingjian Chen, Yusuke Iwasawa, Yutaka Matsuo, Chanjun Park, Rex Ying, Irene Li

llms fine-tuning benchmarks

arXiv Computation and Language

Sep 1

Graph Evidence Is Not Enough: Diagnosing Native Decoder Use in Graph-Augmented LLMs

arXiv:2608.30437v1 Announce Type: new Abstract: Graph-augmented large language models often assume that graph evidence produced by external computation and placed in the input can be used by the nati...

By Xiaoyu Guo, Pengcheng Chen, Jiong Yu, Yi Lu, Yaohua Wang, Ziyang Li

llms safety

One email a morning, machine-written

Related stories

Diagnosing Multi-step Reasoning Failures in Black-box LLMs via Stepwise Confidence Attribution

Reasoning Structure of Large Language Models

Graph Evidence Is Not Enough: Diagnosing Native Decoder Use in Graph-Augmented LLMs

GUT: Quantifying and Optimizing the Reasoning Uncertainty of LLMs via Graph Complexity

Omanic: Towards Step-wise Evaluation of Multi-hop Reasoning in Large Language Models

Graph Evidence Is Not Enough: Diagnosing Native Decoder Use in Graph-Augmented LLMs