HOLogicBench is a 1,379-instance benchmark designed to evaluate higher-order logical reasoning in large language models, focusing on operations over rules and policies rather than just object-level inference. It includes Isabelle/HOL formalizations and proof‑verified answers, and provides solver‑verified reference reasoning paths for fine‑grained evaluation. Across 16 model configurations, average accuracy is 53.86%, with the best reaching 65.99%, and analyses show limited alignment with valid derivations, vulnerability to minimal counterfactual perturbations, and distinct failure patterns across reasoning types.
arXiv:2606. 23238v2 Announce Type: replace Abstract: Logical reasoning is essential for reliable AI, yet existing benchmarks are largely first-order-logic-centric, focusing on object-level deduction over fixed predicates.
HybridProver is a unified framework that combines whole-proof synthesis and tactic-based generation using proof sketches as an intermediate representation. Implemented in Isabelle/HOL, it employs two 7B-scale LLMs trained on optimized Isabelle datasets. On the miniF2F Isabelle benchmark, HybridProver achieved a 73.8% success rate, surpassing the previous state of the art of 61.9%, and ablation studies examined the effects of dataset quality, training settings, and sampling strategies.
arXiv:2606. 20227v1 Announce Type: new Abstract: Large Language Models (LLMs) have made significant progress in reasoning, particularly in deductive reasoning, which is crucial for high-stakes decision-making.
arXiv:2605. 23965v2 Announce Type: replace Abstract: Large Language Models (LLMs) achieve strong performance on logical reasoning benchmarks, yet their reliability remains uncertain.
LGMT is an open‑source tool that tests logical reasoning reliability in large language models by generating logically equivalent follow‑up test cases from a source premise‑conclusion pair and checking for consistency in the model’s entailment predictions. It includes 20 built‑in metamorphic relations, allows custom relations verified by the Z3 theorem prover, and can be run via CLI or a web studio, even offline without an API key. In experiments on six popular LLMs across 3,091 metamorphic groups, LGMT uncovered inconsistency rates between 20.84% and 29.44%.
arXiv:2512. 10187v3 Announce Type: replace Abstract: LLMs excel at reasoning, but validating their steps remains challenging.
arXiv:2601.20055v3 Announce Type: replace-cross Abstract: Despite the syntactic fluency of Large Language Models (LLMs), ensuring their logical correctness in high-stakes domains remains a fundamenta...
LiveMathematicianBench is a dynamic multiple‑choice benchmark for research‑level mathematical reasoning, built from recent arXiv papers published after model training cutoffs. It introduces a thirteen‑category logical taxonomy of theorem types and uses a proof‑sketch‑guided distractor pipeline to create plausible but invalid answer choices, enhancing sensitivity to genuine reasoning. Evaluation shows current large language models perform poorly, with the best model scoring 43.5% overall and only 17.6% under substitution‑resistant conditions, indicating the benchmark’s difficulty and realism.
arXiv:2606. 15258v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly capable of mathematical problem solving and can even assist with research-level proofs, yet we still lack a scalable and reproducible way to measure step-level reasoning in long proofs across diverse sources.
arXiv:2604. 11556v2 Announce Type: replace-cross Abstract: LLM-assisted software development has become increasingly prevalent, and can generate large-scale systems, such as compilers.
arXiv:2603. 05290v2 Announce Type: replace Abstract: Large language models (LLMs) achieve promising performance, yet their ability to reason remains poorly understood.
arXiv:2508. 09125v3 Announce Type: replace-cross Abstract: Instruction following has catalyzed the recent era of Large Language Models (LLMs) and is the foundational skill underpinning more advanced capabilities such as reasoning and agentic behaviors.