The paper proposes a competence‑based evaluation (CBE) framework for assessing logical reasoning in large language models. CBE tests whether models consistently apply the same reasoning competence across a family of problems, using a parameterized generator to vary complexity while preserving the underlying rule. Experiments on GPT and Qwen models show that even strong models often fail to meet the competence condition across the tested family, indicating limitations in generalization.
arXiv:2606. 20227v1 Announce Type: new Abstract: Large Language Models (LLMs) have made significant progress in reasoning, particularly in deductive reasoning, which is crucial for high-stakes decision-making.
arXiv:2604. 11996v2 Announce Type: replace-cross Abstract: Should we trust Large Language Models (LLMs) with high accuracy?
arXiv:2609.38409v1 Announce Type: new Abstract: Recent progress in large language model reasoning has been driven by benchmarks and reinforcement learning environments with automatically verifiable r...
LogicSkills is a benchmark designed to isolate three core logical abilities in large language models: formal symbolization, countermodel construction, and validity assessment. The dataset draws items from the two-variable fragment of first‑order logic without identity, presented in both English and a Carrollian nonce‑word language, and all instances are solver‑verified with Z3. Results show that conventional instruction‑tuned LLMs excel at validity assessment but struggle with symbolization and countermodel construction, whereas recent reasoning‑tuned models perform well across all tasks, indicating a more systematic logical skill profile.
arXiv:2509. 14704v3 Announce Type: replace Abstract: Benchmark saturation and training-data contamination increasingly obscure whether reported gains in large language models (LLMs) reflect genuine advances in reasoning or familiarity with recurring patterns in benchmark problems.
arXiv:2607. 06327v1 Announce Type: cross Abstract: Uncertainty estimation (UE) enables LLM-powered systems to recognize when to abstain, yet existing research has predominantly focused on English.
The paper introduces LSR‑Ben, a benchmark designed to evaluate process reward models (PRMs) on scientific and logical reasoning tasks, addressing a gap left by existing math‑focused benchmarks. Experiments on 22 models reveal that PRMs and LLMs perform poorly in non‑mathematical domains, with LLMs tending to over‑identify errors while PRMs tend to overlook them. LSR‑Ben aims to spur research that broadens PRM applicability and improves LLM reasoning.
PotARCin expands the ARC benchmark by evaluating abstract reasoning across five dimensions—Definition, Classification, Constrained Generation, Editing, and Inversion—using programmatic generation of new task instances. The study shows a 25‑52 percentage‑point performance gap between standard ARC evaluation and PotARCin, and reveals that multi‑dimensional assessment can reorder models that appear equivalent under single‑metric accuracy. Additionally, a new held‑out set, P‑ARC, demonstrates low model accuracy (1‑8%) across all dimensions, highlighting the need for more comprehensive tests of abstract reasoning.
arXiv:2607. 05992v1 Announce Type: cross Abstract: Mathematical reasoning has become a central task for evaluating and tuning reasoning Large Language Models (LLMs), yet existing benchmarks remain heavily biased toward high-resource languages, with English and Chinese dominating both pre-training corpora and evaluation suites.
arXiv:2606. 09157v1 Announce Type: cross Abstract: This paper revisits our pipeline called Syllogistic Evaluation Framework-Common Logic Grammar Construction (SEF-CLGC).
arXiv:2603. 00546v2 Announce Type: replace Abstract: Using Multimodal Large Language Models (MLLMs) as judges to achieve precise and consistent evaluations has gradually become an emerging paradigm across various domains.
arXiv:2606. 26797v1 Announce Type: new Abstract: Supervised fine-tuning (SFT) on a small, high-quality set of long reasoning traces is an effective approach for eliciting strong reasoning capabilities in Large Language Models (LLMs).