arXiv AI
Aug 14
Are you Talking Logic to Me? Assessing Language Models Syllogistic Reasoning Capabilities
arXiv:2608. 12374v1 Announce Type: cross Abstract: Language models (LMs) struggle with logical tasks like reasoning on syllogisms.
SEF-CLGC at SemEval-2026 Task 11: Logical Notation Impact on Language Model Performance
arXiv:2606. 09157v1 Announce Type: cross Abstract: This paper revisits our pipeline called Syllogistic Evaluation Framework-Common Logic Grammar Construction (SEF-CLGC).
Related stories
arXiv AI
Jul 17
HABIB_TAZ at SemEval-2026 Task 11: Disentangling Formal Logic from Content via Synthetic Training and Multi-Objective Optimization
arXiv:2607. 14349v1 Announce Type: cross Abstract: While Large Language Models (LLMs) excel in many general NLP tasks, their formal reasoning capabilities are often compromised by content effects, demonstrating a measurable bias towards real-world plausibility.
arXiv AI
Jun 19
QMFOL: Benchmarking Large Language Model Reasoning via Quantifiable Monadic First-Order Logic Test Case Generation
arXiv:2606. 20227v1 Announce Type: new Abstract: Large Language Models (LLMs) have made significant progress in reasoning, particularly in deductive reasoning, which is crucial for high-stakes decision-making.
arXiv AI
Jul 28
Do Diagrams Help Large Language Models Reason? Evidence from Syllogistic Reasoning
arXiv:2607. 23513v1 Announce Type: cross Abstract: Diagrams are widely used to support logical reasoning, and prior studies suggest that representations such as Euler diagrams can improve human reasoning performance.
arXiv Machine Learning
Jun 16
Pushing the Boundaries of Natural Reasoning: Interleaved Bonus from Formal-Logic Verification
arXiv:2601. 22642v2 Announce Type: replace Abstract: Large Language Models (LLMs) show remarkable capabilities, yet their stochastic next-token prediction creates logical inconsistencies and reward hacking that formal symbolic systems avoid.
arXiv Machine Learning
Aug 31
Beyond Output Correctness: Benchmarking and Evaluating Large Language Model Reasoning in Coding Tasks
The paper introduces CodeRQ-Bench, the first benchmark for assessing large language model reasoning quality across coding tasks such as generation, summarization, and classification. It analyzes over a thousand mismatches from existing evaluators, identifies recurring limitations, and derives design insights that lead to a new two‑stage evaluator, VERA. Experiments show VERA outperforms strong baselines, improving AUCROC by up to 0.26 and AUPRC by up to 0.21 on four datasets.
Hugging Face Trending Papers
Jul 26
Do Diagrams Help Large Language Models Reason? Evidence from Syllogistic Reasoning
Diagrams are widely used to support logical reasoning, and prior studies suggest that representations such as Euler diagrams can improve human reasoning performance. Recent work has also explored their effects on large language models (LLMs).
arXiv AI
Sep 7
Harnessing the Reasoning Economy: A Survey of Efficient Reasoning for Large Language Models
The paper surveys efficient reasoning in large language models, contrasting fast intuitive (System 1) and slow deep (System 2) reasoning. It analyzes why System 2 is computationally costly yet more accurate, and why System 1 is efficient but less effective. The survey covers causes of inefficiency, patterns of reasoning behavior, and potential solutions to balance performance and computational budgets, offering actionable insights and an open‑source repository for ongoing research.
arXiv Computation and Language
4d ago
Think Multilingual, Not Harder: A Data-Efficient Framework for Teaching Reasoning Models to Code-Switch
arXiv:2604.15490v2 Announce Type: replace Abstract: Recent developments in reasoning capabilities have enabled large language models to solve increasingly complex mathematical, symbolic, and logical...
arXiv Machine Learning
Jul 21
LogicIF: Towards Complex Logic Instruction Following
arXiv:2508. 09125v3 Announce Type: replace-cross Abstract: Instruction following has catalyzed the recent era of Large Language Models (LLMs) and is the foundational skill underpinning more advanced capabilities such as reasoning and agentic behaviors.
arXiv Computation and Language
Sep 1
Reasoning over Grammar: Can Synthetic Linguistic Reasoning Traces Enhance Low-Resource Machine Translation?
The paper explores whether structured linguistic reasoning traces can improve low‑resource machine translation by guiding large language models (LLMs). It proposes a pipeline that automatically generates step‑by‑step reasoning traces from Universal Dependencies treebanks, dictionaries, and grammar‑rule banks, and evaluates these traces in in‑context learning, supervised fine‑tuning, and reinforcement fine‑tuning on Xibe and Chintang. The results show that providing reliable reasoning traces at inference time significantly boosts translation quality, whereas using them as training data yields smaller, less consistent gains, indicating that LLMs can benefit from grammatical guidance but struggle to generate accurate analyses themselves.
arXiv AI
Jun 3
Reasoning Structure of Large Language Models
arXiv:2606. 03883v1 Announce Type: new Abstract: Large reasoning models (LRMs) are often evaluated using metrics such as final-answer accuracy or token count.