arXiv:2605. 23965v2 Announce Type: replace Abstract: Large Language Models (LLMs) achieve strong performance on logical reasoning benchmarks, yet their reliability remains uncertain.
By Zenghui Zhou, Man Li, Xiaoke Fang, Xinyi Zhou, Weibin Lin, Zheng Zheng
Detecting Inconsistencies in Model Specifications with LLM-as-Verifier Reasoning introduces VeriSpec, a method that audits natural‑language model specifications directly using an LLM as a verifier. VeriSpec extracts structured rules, builds a topic‑guided graph to cluster related rules, and applies LLM‑based reasoning to find inconsistencies. In a case study on the OpenAI Model Spec, it identified 405 rules, validated five inconsistencies, and outperformed five baselines with the highest precision and lowest cost per validated inconsistency.
By Zichen Xie, Mrigank Pawagi, Lize Shao, Yang Hu, Wenxi Wang
arXiv:2606. 20227v1 Announce Type: new Abstract: Large Language Models (LLMs) have made significant progress in reasoning, particularly in deductive reasoning, which is crucial for high-stakes decision-making.
By Xinyi Zheng, Ling Shi, Tianlong Yu, Yongxin Zhao, Lorenz Goette, Kailong Wang
HOLogicBench is a 1,379-instance benchmark designed to evaluate higher-order logical reasoning in large language models, focusing on operations over rules and policies rather than just object-level inference. It includes Isabelle/HOL formalizations and proof‑verified answers, and provides solver‑verified reference reasoning paths for fine‑grained evaluation. Across 16 model configurations, average accuracy is 53.86%, with the best reaching 65.99%, and analyses show limited alignment with valid derivations, vulnerability to minimal counterfactual perturbations, and distinct failure patterns across reasoning types.
By Yucheng Wu, Jundong Xu, Mingzhen Ju, Yue Yu, Chenpeng Wang, Haoxuan Li, Liangming Pan
arXiv:2608.30581v1 Announce Type: new
Abstract: Large language models (LLMs) are used as post hoc explainers of sequential decision-making policies, producing natural-language explanations of why an...
By Dennis Gross, Helge Spieker
The paper introduces SWE-Flux, a repository‑level benchmark designed to test large language models’ ability to reason about runtime behavior. It contains 480 execution‑grounded instances from 12 real Python repositories, with gold answers automatically harvested from instrumented test executions. Evaluation of five LLMs shows the task remains difficult, with the best model achieving only 37% accuracy, and the benchmark can generate challenging variants through input perturbation.
By Hamed Taherkhani, Mohammad Abdollahi, Melika Sepidband, Hridya Dhulipala, Tien N. Nguyen, Hadi Hemmati