LGMT is an open‑source tool that tests logical reasoning reliability in large language models by generating logically equivalent follow‑up test cases from a source premise‑conclusion pair and checking for consistency in the model’s entailment predictions. It includes 20 built‑in metamorphic relations, allows custom relations verified by the Z3 theorem prover, and can be run via CLI or a web studio, even offline without an API key. In experiments on six popular LLMs across 3,091 metamorphic groups, LGMT uncovered inconsistency rates between 20.84% and 29.44%.
arXiv:2605. 23965v2 Announce Type: replace Abstract: Large Language Models (LLMs) achieve strong performance on logical reasoning benchmarks, yet their reliability remains uncertain.
Detecting Inconsistencies in Model Specifications with LLM-as-Verifier Reasoning introduces VeriSpec, a method that audits natural‑language model specifications directly using an LLM as a verifier. VeriSpec extracts structured rules, builds a topic‑guided graph to cluster related rules, and applies LLM‑based reasoning to find inconsistencies. In a case study on the OpenAI Model Spec, it identified 405 rules, validated five inconsistencies, and outperformed five baselines with the highest precision and lowest cost per validated inconsistency.
arXiv:2606. 20227v1 Announce Type: new Abstract: Large Language Models (LLMs) have made significant progress in reasoning, particularly in deductive reasoning, which is crucial for high-stakes decision-making.
HOLogicBench is a 1,379-instance benchmark designed to evaluate higher-order logical reasoning in large language models, focusing on operations over rules and policies rather than just object-level inference. It includes Isabelle/HOL formalizations and proof‑verified answers, and provides solver‑verified reference reasoning paths for fine‑grained evaluation. Across 16 model configurations, average accuracy is 53.86%, with the best reaching 65.99%, and analyses show limited alignment with valid derivations, vulnerability to minimal counterfactual perturbations, and distinct failure patterns across reasoning types.
arXiv:2608.30581v1 Announce Type: new Abstract: Large language models (LLMs) are used as post hoc explainers of sequential decision-making policies, producing natural-language explanations of why an...
The paper introduces SWE-Flux, a repository‑level benchmark designed to test large language models’ ability to reason about runtime behavior. It contains 480 execution‑grounded instances from 12 real Python repositories, with gold answers automatically harvested from instrumented test executions. Evaluation of five LLMs shows the task remains difficult, with the best model achieving only 37% accuracy, and the benchmark can generate challenging variants through input perturbation.
ORSMT is a tool that uses Satisfiability Modulo Theories to verify and repair artifacts produced by large language models when automating optimization modeling from natural language. It provides a typed semantic interface, exact SMT relation queries, and mechanisms to align decisions, project auxiliary variables, and reconstruct artifacts, thereby improving model-code match rates across thousands of problems. The system demonstrates significant gains in correctness and matching rates compared to existing methods such as ReLoop and Opt‑Verifier.
The paper introduces DRAGO, a framework that repairs formal specifications generated by large language models (LLMs) using deterministic, formalism‑specific diagnosis instead of ungrounded LLM critiques. DRAGO selectively rewrites specifications only when the diagnosed error weight decreases, reducing regressions from 11.8% to 1.3% and improving answer accuracy by 12.9 percentage points over zero‑shot formalization and 8.7 percentage points over LLM‑diagnostic repair across five benchmarks. Additionally, the authors identified and corrected rendering defects in Planetarium that caused half of its floor‑tile split instances to be inconsistent with reference formalizations.
arXiv:2607. 23019v1 Announce Type: new Abstract: Chain-of-thought (CoT) prompting enables large language models (LLMs) to tackle multi-step reasoning tasks, yet the generated intermediate steps are not guaranteed to be logically sound.
Large language models (LLMs) are used as post hoc explainers of sequential decision-making policies, producing natural-language explanations of why an action was chosen. However, LLMs often generate p...
The paper introduces LinguaArray, a framework that uses large language models (LLMs) to suggest a small set of focal ground terms or constraints—called a Semantic Focus Set—in quantified array SMT solving. The LLM’s suggestions are then verified by a conventional SMT solver, which remains the sole source of proof. Experiments on 1,015 SMT-LIB instances show that LinguaArray solves up to 69.1% of cases and increases the number of solved instances by 105%–397% compared to the backends alone.
arXiv:2608.28725v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used as graders, verifiers, and process auditors, but most mathematical evaluations still emphasize final...