arXiv:2606. 16603v1 Announce Type: cross Abstract: LLM-based agents have demonstrated strong capabilities in data-intensive analytical tasks, yet their outputs are rarely verifiable: a reliance on linear text trajectories makes their reasoning difficult to audit.
By Jiajie Jin, Zhao Yang, Wenle Liao, Yuyang Hu, Guanting Dong, Xiaoxi Li, Yutao Zhu, Zhicheng Dou
The paper introduces a typed reasoning graph framework to compare human and large language model (LLM) reasoning paths in scientific fact‑checking. By modeling explanations as graphs linking false claims to study context, findings, premises, and fallacy labels, the authors enable one‑to‑one alignment of human and LLM reasoning at the sub‑graph level. Using 84 false claims from MISSCIPLUS, they evaluate GPT‑5, Claude Opus 4.7, and Qwen3‑32B, finding distinct performance patterns: Qwen3‑32B has the lowest verdict failure rate, GPT‑5 shows the highest human alignment, and Claude Opus 4.7, while weak at verdict prediction, often produces valid reasoning in successful cases.
By Abdul Ghafoor, Muhammad Arslan Manzoor, Yufang Hou
The paper introduces RuVerBench, a benchmark with 2,458 instances for evaluating the reliability of Large Language Models acting as judges (LaaJ) in verifying rubric compliance within agentic scenarios such as deep research and agentic coding. It systematically meta‑evaluates frontier LLMs, revealing that even the most advanced models perform well yet still produce substantial noise. The study also examines how prompt design, batching, and majority voting affect verification accuracy, noting that weaker models are more prompt‑sensitive, batched verification trades accuracy for efficiency, and majority voting offers diminishing returns.
By Yangda Peng, Yunjia Qi, Haotian Xia, Guanzhong He, Xintong Shi, Richeng Xuan, Songyuanyi Lu, Yixian Liu, Zhichao Hu, Yuhong Liu, Hao Peng
The paper introduces Reasoning DAG Probing, a method that treats premises, intermediate conclusions, and final answers as nodes in a directed acyclic graph (DAG). By training lightweight probes on LLM hidden states, the authors show that DAG structure is encoded in the models, peaking in intermediate layers and becoming stronger with larger models. During autoregressive generation, the latent DAG structure emerges progressively, anticipating answer correctness and enabling correction of some incorrect outputs by steering activations toward the ground‑truth DAG.
By Tianjun Zhong, Linyang He, Ziyang Li, Nima Mesgarani
The paper introduces a two-stage diagnostic evaluation to assess the structural reliability of reasoning graphs used in large language models for tasks like multi-hop question answering. Stage 1 examines how graph reliability depends on topology, prompt design, and model family, while Stage 2 evaluates the impact of graph errors on uncertainty estimation. Findings show that graph-based computation is vulnerable to upstream errors, offers limited benefit as prompt context alone, but improves confidence-based answer discrimination and enables node-level diagnostics.
By Xinyu Liu, Dongqi Fu, Jun Wu
The paper presents a unified dataset of 350 Lean tasks to evaluate specification generation (SpecGen) for verifiable code, combining metrics of formal validity, reference similarity, equivalence, and behavioural adequacy. It highlights that restricting similarity measures to a subset of tasks can shift rankings, and shows that a perfect postcondition score can still miss critical input coverage, underscoring the need for separate feedback on input and output constraints.
By Srijith Nair, Aditya Vempaty, Jia Liu, Ashish Jagmohan