The paper proposes a competence‑based evaluation (CBE) framework for assessing logical reasoning in large language models. CBE tests whether models consistently apply the same reasoning competence across a family of problems, using a parameterized generator to vary complexity while preserving the underlying rule. Experiments on GPT and Qwen models show that even strong models often fail to meet the competence condition across the tested family, indicating limitations in generalization.
By Jize Jiang, Chengxiang Zhai
arXiv:2609.38409v1 Announce Type: new
Abstract: Recent progress in large language model reasoning has been driven by benchmarks and reinforcement learning environments with automatically verifiable r...
By \.Ibrahim Ethem Deveci, Funda Tan \c{C}al{\i}k, Bar{\i}\c{s} Deniz Sa\u{g}lam, Duygu Ataman
arXiv:2509. 14704v3 Announce Type: replace Abstract: Benchmark saturation and training-data contamination increasingly obscure whether reported gains in large language models (LLMs) reflect genuine advances in reasoning or familiarity with recurring patterns in benchmark problems.
By Masaharu Mizumoto, Dat Nguyen, Zhiheng Han, Xingfu Li, Yo Nakawake, Le Minh Nguyen
arXiv:2606. 20227v1 Announce Type: new Abstract: Large Language Models (LLMs) have made significant progress in reasoning, particularly in deductive reasoning, which is crucial for high-stakes decision-making.
By Xinyi Zheng, Ling Shi, Tianlong Yu, Yongxin Zhao, Lorenz Goette, Kailong Wang
arXiv:2608. 03291v1 Announce Type: cross Abstract: Chain-of-thought (CoT) reasoning improves large language model (LLM) performance while also providing an observable interface to the model's reasoning process.
By Shashwat Sourav, Aishwarya Balwani
arXiv:2606. 08471v1 Announce Type: cross Abstract: Recently, language models have made rapid progress across various domains and applications.
By Marina Igitkhanian, Erik Arakelyan