arXiv AI
Aug 5
Shorter Reasoning, Earlier Answers? An Evaluation of Reasoning Interfaces
arXiv:2608. 03401v1 Announce Type: cross Abstract: Large language models often reason at length before answering, increasing cost and latency.
Distinguishing among reasoning levels of GPT 5.5 Pro via a grammar lens benchmark
The paper introduces a grammar‑lens benchmark that generates data at progressively higher computational levels to test language models. Using this benchmark, GPT‑5.5 and its Pro variants were evaluated, showing increasing acceptance rates of 16%, 40%, 43%, and 52% as the models’ test‑time reasoning levels rose. The study also notes that higher compute effort leads to responses that diverge more from human answers, hinting at a potential capability‑versus‑alignment tension.
Related stories
arXiv AI
Oct 5
SyntaxBench: A Statistical Diagnostic Framework for Character-Level Reasoning in Large Language Models
SyntaxBench is a diagnostic benchmark and statistical evaluation framework for character‑level reasoning in large language models, comprising five core tasks—character counting, letter containment, palindrome detection, edit distance, and longest‑string selection—and a harder substring‑extraction stress test called index_to_span. The benchmark uses paired English and random‑string inputs, zero‑, one‑, and four‑shot prompts, and evaluates models from 2B to 32B parameters across multiple reasoning modes. It reports a wide range of metrics, including exact‑match and relaxed accuracy, Cohen’s kappa, McNemar tests, bootstrap confidence intervals, Kendall’s tau, class‑conditional metrics, tokenization analysis, and multiple‑comparison‑corrected tests.
arXiv AI
Sep 10
LogicSkills: A Structured Benchmark for Formal Reasoning in Large Language Models
LogicSkills is a benchmark designed to isolate three core logical abilities in large language models: formal symbolization, countermodel construction, and validity assessment. The dataset draws items from the two-variable fragment of first‑order logic without identity, presented in both English and a Carrollian nonce‑word language, and all instances are solver‑verified with Z3. Results show that conventional instruction‑tuned LLMs excel at validity assessment but struggle with symbolization and countermodel construction, whereas recent reasoning‑tuned models perform well across all tasks, indicating a more systematic logical skill profile.
arXiv AI
Oct 19
Competence-Based Evaluation of Logical Reasoning in Large Language Models
The paper proposes a competence‑based evaluation (CBE) framework for assessing logical reasoning in large language models. CBE tests whether models consistently apply the same reasoning competence across a family of problems, using a parameterized generator to vary complexity while preserving the underlying rule. Experiments on GPT and Qwen models show that even strong models often fail to meet the competence condition across the tested family, indicating limitations in generalization.
arXiv AI
Jul 8
The relationship between reasoning and performance in large language models--o3 (mini) thinks harder, not longer
arXiv:2502. 15631v2 Announce Type: replace-cross Abstract: Large language models have demonstrated remarkable progress in mathematical reasoning, leveraging chain-of-thought and reinforcement learning.
arXiv Machine Learning
Aug 5
M-GATE: Multilingual Grammar, Accuracy in Translation, and Efficiency Benchmark for Large Language Models
arXiv:2608. 03803v1 Announce Type: cross Abstract: Multilingual language models are deployed across a hundred or more languages, yet most benchmarks test whether a model can perform a task _in_ a language rather than whether it commands the language itself, conflating fluency with proficiency.
arXiv AI
Jul 17
Simplicity Paradox: Debunking myths about prompting and datasets for LLM evaluation
arXiv:2607. 14109v1 Announce Type: cross Abstract: Probing the capabilities of Large Language Models (LLMs) and building robust solutions for Multiple-Choice Question Answering (MCQA) remain central challenges in natural language understanding.
arXiv AI
Jun 9
More Yap Less Meaning: Uncovering Self-Improvement Behavior in SLMs
arXiv:2606. 08471v1 Announce Type: cross Abstract: Recently, language models have made rapid progress across various domains and applications.
arXiv AI
Aug 26
Recursive Agentic Reasoning
The paper proposes a unified framework for test‑time reasoning methods, framing them as recursion operators—GROW, PRUNE, and BRANCH—applied to an agent’s reasoning trace. Experiments across five benchmarks and three frontier models show that BRANCH, which samples and selects among multiple reasoning paths, consistently outperforms the other operators and a single‑pass chain‑of‑thought baseline, improving accuracy by an average of 5.98 percentage points. The study also highlights the importance of paired evaluation and careful handling of scoring‑pipeline failures, as these factors can significantly alter comparative outcomes.
arXiv AI
Aug 26
Evaluating Language Models on Cross-Language Code Functional Equivalence
The paper introduces PolyHuman, a dataset of human-written programs in C++, Java, and Python, to test whether large language models can judge functional equivalence across languages. Using this dataset, the authors evaluate several open-weight and proprietary LLMs, finding that models struggle more with harder problems, show language-specific biases, and rely partly on superficial similarity cues. They also observe run‑to‑run instability in GPT‑o4‑mini, concluding that current LLMs do not reliably capture functional equivalence within or across programming languages.
arXiv AI
Jun 29
CalBrief: A Pilot Diagnostic Benchmark for Evidence-Calibrated Scientific Briefing with Large Language Models
arXiv:2606. 27383v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used as research assistants, yet it remains unclear whether they can calibrate research takeaways to the strength and scope of the supporting evidence.
arXiv AI
Sep 2
The Importance of Being Statistically Earnest: A Critical Re-evaluation of GSM-Symbolic
The paper re‑examines the GSM‑Symbolic benchmark, arguing that its claim of widespread reasoning failures in 25 LLMs is based on weak statistics. Using bootstrapped Generalised Linear Mixed Models on 20 open‑weight models, only eight show significant performance changes, and a systematic shift toward larger integers in the dataset explains many of these effects. The authors also uncover model‑specific failure modes such as variable binding fragility, arithmetic limits, and dual‑task interference, cautioning against blanket conclusions about LLM reasoning.