arXiv AI

QMFOL: Benchmarking Large Language Model Reasoning via Quantifiable Monadic First-Order Logic Test Case Generation

arXiv:2606. 20227v1 Announce Type: new Abstract: Large Language Models (LLMs) have made significant progress in reasoning, particularly in deductive reasoning, which is crucial for high-stakes decision-making.

arXiv Machine Learning

Jul 21

LogicIF: Towards Complex Logic Instruction Following

arXiv:2508. 09125v3 Announce Type: replace-cross Abstract: Instruction following has catalyzed the recent era of Large Language Models (LLMs) and is the foundational skill underpinning more advanced capabilities such as reasoning and agentic behaviors.

By Mian Zhang, Shujian Liu, Sixun Dong, Ming Yin, Yebowen Hu, Xun Wang, Simin Ma, Song Wang, Sathish Reddy Indurthi, Haoyun Deng, Zhiyu Zoey Chen, Kaiqiang Song

llms agents benchmarks

arXiv AI

Jul 21

SATQuest: A Verifier for Logical Reasoning Evaluation and Reinforcement Fine-Tuning of LLMs

arXiv:2509. 00930v2 Announce Type: replace Abstract: Large language models (LLMs) exhibit strong general reasoning, yet the community lacks controllable, scalable, and verifiable tools to analyze and improve these abilities.

By Yanxiao Zhao, Yaqian Li, Zihao Bo, Rinyoichi Takezoe, Haojia Hui, Mo Guang, Lei Ren, Xiaolin Qin, Kaiwen Long

llms fine-tuning benchmarks

arXiv AI

Jun 9

LGMT: Logic-Grounded Metamorphic Testing for Evaluating the Reasoning Reliability of LLMs

arXiv:2605. 23965v2 Announce Type: replace Abstract: Large Language Models (LLMs) achieve strong performance on logical reasoning benchmarks, yet their reliability remains uncertain.

By Zenghui Zhou, Man Li, Xiaoke Fang, Xinyi Zhou, Weibin Lin, Zheng Zheng

llms benchmarks

arXiv Machine Learning

Jun 16

Pushing the Boundaries of Natural Reasoning: Interleaved Bonus from Formal-Logic Verification

arXiv:2601. 22642v2 Announce Type: replace Abstract: Large Language Models (LLMs) show remarkable capabilities, yet their stochastic next-token prediction creates logical inconsistencies and reward hacking that formal symbolic systems avoid.

By Chuxue Cao, Jinluan Yang, Haoran Li, Kunhao Pan, Zijian Zhao, Zhengyu Chen, Yuchen Tian, Lijun Wu, Conghui He, Sirui Han, Yike Guo

llms fine-tuning benchmarks

arXiv AI

Jun 16

Towards Advanced Mathematical Reasoning for LLMs via First-Order Logic Theorem Proving

arXiv:2506. 17104v2 Announce Type: replace Abstract: Large language models (LLMs) have shown promising first-order logic (FOL) reasoning capabilities with applications in various areas.

By Chuxue Cao, Mengze Li, Juntao Dai, Jinluan Yang, Zijian Zhao, Shengyu Zhang, Weijie Shi, Chengzhong Liu, Sirui Han, Yike Guo

llms benchmarks

arXiv AI

Jun 3

Reasoning Structure of Large Language Models

arXiv:2606. 03883v1 Announce Type: new Abstract: Large reasoning models (LRMs) are often evaluated using metrics such as final-answer accuracy or token count.

By Fr\'ed\'eric Berdoz, Luca A. Lanzend\"orfer, Fabian Farestam, Roger Wattenhofer

llms benchmarks

arXiv AI

Aug 5

TQLite: Multi-LLM Jury Guided Distillation for Real-time MQM Translation Quality Evaluation

arXiv:2608. 02975v1 Announce Type: cross Abstract: Large language models (LLMs) have demonstrated impressive performance in MQM-based translation quality (TQ) evaluation, and recent advances in large reasoning models (LRMs) promise even greater improvements.

By Bhavin Jawade, Cameron R. Wolfe

llms efficiency benchmarks

arXiv AI

Jun 24

HOLMES: Evaluating Higher-Order Logical Reasoning in LLMs

arXiv:2606. 23238v2 Announce Type: replace Abstract: Logical reasoning is essential for reliable AI, yet existing benchmarks are largely first-order-logic-centric, focusing on object-level deduction over fixed predicates.

By Yucheng Wu, Jundong Xu, Mingzhen Ju, Yue Yu, Chenpeng Wang, Haoxuan Li, Liangming Pan

llms benchmarks

arXiv AI

Jul 8

Mathematical Reasoning in Large Language Models: Benchmarks, Architectures, Evaluation, and Open Challenges

arXiv:2605. 19723v2 Announce Type: replace-cross Abstract: Mathematical reasoning is essential for problem-solving in education, science, and industry, serving as a crucial benchmark for evaluating artificial intelligence systems.

By Husnain Amjad, Raja Khurram Shahzad, Aamir Shahzad, Mehwish Fatima

llms fine-tuning benchmarks

arXiv AI

Jun 9

SEF-CLGC at SemEval-2026 Task 11: Logical Notation Impact on Language Model Performance

arXiv:2606. 09157v1 Announce Type: cross Abstract: This paper revisits our pipeline called Syllogistic Evaluation Framework-Common Logic Grammar Construction (SEF-CLGC).

By Hanna Abi Akl, Fabien Gandon, Catherine Faron, Pierre Monnin

llms safety

arXiv AI

Jul 3

ADVENT: LLM-Driven Automatic Predicate Invention for ILP

arXiv:2607. 01585v1 Announce Type: cross Abstract: Predicate invention (PI), the creation of new predicates to extend the hypothesis space, remains a critical bottleneck in Inductive Logic Programming (ILP).

By Tingting Yu, Pei-Cing Huang, Chan Hsu, Chan-Tung Ku, Yihuang Kang

llms

arXiv AI

5d ago

Are you Talking Logic to Me? Assessing Language Models Syllogistic Reasoning Capabilities

arXiv:2608. 12374v1 Announce Type: cross Abstract: Language models (LMs) struggle with logical tasks like reasoning on syllogisms.

By Hanna Abi Akl, Fabien Gandon, Catherine Faron, Pierre Monnin

llms fine-tuning

Related stories