arXiv AI October 28, 2026 By Yucheng Wu, Jundong Xu, Mingzhen Ju, Yue Yu, Chenpeng Wang, Haoxuan Li, Liangming Pan

HOLogicBench: Evaluating Higher-Order Logical Reasoning in LLMs

HOLogicBench is a 1,379-instance benchmark designed to evaluate higher-order logical reasoning in large language models, focusing on operations over rules and policies rather than just object-level inference. It includes Isabelle/HOL formalizations and proof‑verified answers, and provides solver‑verified reference reasoning paths for fine‑grained evaluation. Across 16 model configurations, average accuracy is 53.86%, with the best reaching 65.99%, and analyses show limited alignment with valid derivations, vulnerability to minimal counterfactual perturbations, and distinct failure patterns across reasoning types.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI

Jun 24

HOLMES: Evaluating Higher-Order Logical Reasoning in LLMs

arXiv:2606. 23238v2 Announce Type: replace Abstract: Logical reasoning is essential for reliable AI, yet existing benchmarks are largely first-order-logic-centric, focusing on object-level deduction over fixed predicates.

By Yucheng Wu, Jundong Xu, Mingzhen Ju, Yue Yu, Chenpeng Wang, Haoxuan Li, Liangming Pan

llms benchmarks

arXiv AI

Aug 28

HybridProver: Augmenting Theorem Proving with LLM-Driven Proof Synthesis and Refinement

HybridProver is a unified framework that combines whole-proof synthesis and tactic-based generation using proof sketches as an intermediate representation. Implemented in Isabelle/HOL, it employs two 7B-scale LLMs trained on optimized Isabelle datasets. On the miniF2F Isabelle benchmark, HybridProver achieved a 73.8% success rate, surpassing the previous state of the art of 61.9%, and ablation studies examined the effects of dataset quality, training settings, and sampling strategies.

By Jilin Hu, Jianyu Zhang, Yongwang Zhao, Talia Ringer

llms benchmarks

arXiv AI

Jun 19

QMFOL: Benchmarking Large Language Model Reasoning via Quantifiable Monadic First-Order Logic Test Case Generation

arXiv:2606. 20227v1 Announce Type: new Abstract: Large Language Models (LLMs) have made significant progress in reasoning, particularly in deductive reasoning, which is crucial for high-stakes decision-making.

By Xinyi Zheng, Ling Shi, Tianlong Yu, Yongxin Zhao, Lorenz Goette, Kailong Wang

llms benchmarks

arXiv AI

Jun 9

LGMT: Logic-Grounded Metamorphic Testing for Evaluating the Reasoning Reliability of LLMs

arXiv:2605. 23965v2 Announce Type: replace Abstract: Large Language Models (LLMs) achieve strong performance on logical reasoning benchmarks, yet their reliability remains uncertain.

By Zenghui Zhou, Man Li, Xiaoke Fang, Xinyi Zhou, Weibin Lin, Zheng Zheng

llms benchmarks

arXiv AI

6d ago

LGMT: A Tool for Testing the Logical Reasoning Reliability of LLMs

LGMT is an open‑source tool that tests logical reasoning reliability in large language models by generating logically equivalent follow‑up test cases from a source premise‑conclusion pair and checking for consistency in the model’s entailment predictions. It includes 20 built‑in metamorphic relations, allows custom relations verified by the Z3 theorem prover, and can be run via CLI or a web studio, even offline without an API key. In experiments on six popular LLMs across 3,091 metamorphic groups, LGMT uncovered inconsistency rates between 20.84% and 29.44%.

By Zenghui Zhou, Xiaoke Fang, Zheng Zheng

llms benchmarks

arXiv Machine Learning

Jun 25

MINIF2F-DAFNY: LLM-Guided Mathematical Theorem Proving via Auto-Active Verification

arXiv:2512. 10187v3 Announce Type: replace Abstract: LLMs excel at reasoning, but validating their steps remains challenging.

By Mantas Baksys, Stefan Zetzsche, Olivier Bouissou, Sean B. Holden

llms benchmarks

One email a morning, machine-written

Related stories

HOLMES: Evaluating Higher-Order Logical Reasoning in LLMs

HybridProver: Augmenting Theorem Proving with LLM-Driven Proof Synthesis and Refinement

QMFOL: Benchmarking Large Language Model Reasoning via Quantifiable Monadic First-Order Logic Test Case Generation

LGMT: Logic-Grounded Metamorphic Testing for Evaluating the Reasoning Reliability of LLMs

LGMT: A Tool for Testing the Logical Reasoning Reliability of LLMs

MINIF2F-DAFNY: LLM-Guided Mathematical Theorem Proving via Auto-Active Verification