arXiv AI
MGSM-Pro: A Simple Strategy for Robust Multilingual Mathematical Reasoning Evaluation
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Jul 8
PluraMath: Extending Mathematical Reasoning Evaluation Beyond High-Resource Languages
arXiv:2607. 05992v1 Announce Type: cross Abstract: Mathematical reasoning has become a central task for evaluating and tuning reasoning Large Language Models (LLMs), yet existing benchmarks remain heavily biased toward high-resource languages, with English and Chinese dominating both pre-training corpora and evaluation suites.
Hugging Face Trending Papers
Jul 14
GSM-Plus-BN: A Perturbation-Based Benchmark for Bangla Mathematical Reasoning in Large Language Models
The evaluation of mathematical reasoning in large language models (LLMs) has predominantly focused on high-resource languages like English. This has created a significant barrier to the equitable development and deployment of AI in linguistically diverse regions such as Bangladesh, where over 230 million people speak Bengali.
arXiv AI
Sep 2
The Importance of Being Statistically Earnest: A Critical Re-evaluation of GSM-Symbolic
The paper re‑examines the GSM‑Symbolic benchmark, arguing that its claim of widespread reasoning failures in 25 LLMs is based on weak statistics. Using bootstrapped Generalised Linear Mixed Models on 20 open‑weight models, only eight show significant performance changes, and a systematic shift toward larger integers in the dataset explains many of these effects. The authors also uncover model‑specific failure modes such as variable binding fragility, arithmetic limits, and dual‑task interference, cautioning against blanket conclusions about LLM reasoning.
arXiv AI
Jun 24
Benchmarking LLMs' Mathematical Reasoning with Unseen Random Variables Questions
arXiv:2501. 11790v5 Announce Type: replace-cross Abstract: Recent studies have raised significant concerns regarding the reliability of current mathematics benchmarks, highlighting issues such as simplistic design and potential data contamination.
arXiv AI
Jul 14
Cost of Reasoning in non-English Languages: A Case Study on Japanese
arXiv:2607. 10114v1 Announce Type: cross Abstract: Reasoning Language Models (RLMs) achieve their strongest performance when they reason in English, the language for which reasoning-oriented training data is most abundant.
arXiv Computation and Language
Sep 10
Building Multilingual Bridges: Data Mixing as the Pillar of Generalization for In-Language Reasoning
arXiv:2609.10445v1 Announce Type: new Abstract: Reasoning language models have made substantial advances on a variety of complex tasks, yet their capabilities remain overwhelmingly English-centric: m...