arXiv AI September 2, 2026 By Dominika Agnieszka D{\l}ugosz, Arlindo Oliveira, Natalia D\'iaz-Rodr\'iguez

The Importance of Being Statistically Earnest: A Critical Re-evaluation of GSM-Symbolic

The paper re‑examines the GSM‑Symbolic benchmark, arguing that its claim of widespread reasoning failures in 25 LLMs is based on weak statistics. Using bootstrapped Generalised Linear Mixed Models on 20 open‑weight models, only eight show significant performance changes, and a systematic shift toward larger integers in the dataset explains many of these effects. The authors also uncover model‑specific failure modes such as variable binding fragility, arithmetic limits, and dual‑task interference, cautioning against blanket conclusions about LLM reasoning.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Hugging Face Trending Papers

Jul 14

GSM-Plus-BN: A Perturbation-Based Benchmark for Bangla Mathematical Reasoning in Large Language Models

The evaluation of mathematical reasoning in large language models (LLMs) has predominantly focused on high-resource languages like English. This has created a significant barrier to the equitable development and deployment of AI in linguistically diverse regions such as Bangladesh, where over 230 million people speak Bengali.

llms benchmarks

arXiv AI

Jul 22

Structured Synthetic Reasoning Data for Arithmetic Fine-Tuning of Small Language Models

arXiv:2607. 18266v1 Announce Type: new Abstract: Small language models are attractive for local deployment, but they often struggle with multi-step arithmetic reasoning.

By Jake O'Grady, Effirul Ramlan

llms fine-tuning benchmarks

arXiv Machine Learning

Jul 2

Clotho: Measuring Task-Specific Pre-Generation Test Adequacy for LLM Inputs

arXiv:2509. 17314v4 Announce Type: replace-cross Abstract: Software increasingly relies on the emergent capabilities of Large Language Models (LLMs), from natural language understanding to program analysis and generation.

By Juyeon Yoon, Somin Kim, Robert Feldt, Shin Yoo

llms benchmarks

arXiv AI

Aug 25

More Accurate or More Efficient? Evaluating Locally Deployed Compact Open-Weight Language Models for Mathematical Reasoning

arXiv:2608.22048v1 Announce Type: new Abstract: Large language models are increasingly deployed on local hardware for privacy, cost, and accessibility reasons. Yet many evaluations emphasize accuracy...

By Orion Powers, Daniella Seum, Khaled Slhoub

llms safety

arXiv AI

Aug 25

Measuring Stability and Failure Behavior in Language Models Under Structured Perturbations

arXiv:2608.22138v1 Announce Type: new Abstract: Language models are usually judged by a single accuracy score, which does not reveal how their performance degrades as inputs are perturbed. We present...

By Samira Golsefid

llms

arXiv AI

Jul 14

Valid $\ne$ Necessary: Diagnosing Latent Inefficiency in Chain-of-Thought

arXiv:2607. 11266v1 Announce Type: new Abstract: Chain-of-Thought (CoT) prompting has significantly advanced the reasoning capabilities of Large Language Models (LLMs), yet it often incurs substantial computational costs due to over-reasoning: the generation of redundant, verbose, or irrelevant steps.

By Daeyeop Lee, Hwanjo Yu

llms efficiency benchmarks

One email a morning, machine-written

Related stories

GSM-Plus-BN: A Perturbation-Based Benchmark for Bangla Mathematical Reasoning in Large Language Models

Structured Synthetic Reasoning Data for Arithmetic Fine-Tuning of Small Language Models

Clotho: Measuring Task-Specific Pre-Generation Test Adequacy for LLM Inputs

More Accurate or More Efficient? Evaluating Locally Deployed Compact Open-Weight Language Models for Mathematical Reasoning

Measuring Stability and Failure Behavior in Language Models Under Structured Perturbations

Valid $\ne$ Necessary: Diagnosing Latent Inefficiency in Chain-of-Thought