← Back to all news
arXiv AI By Kenneth Enevoldsen, Riley Herchert, Sofie Mosegaard, Dan Saattrup Smart, Simon Enni, Isaac Chung, Sofie Bruun, Ayush Sunil Munot, Max M\"uller-Eberstein, Adnan El-Assadi, Elisa Bassignana, Gianluca Barmina, Hafsteinn Einarsson, Iben Nyholm Debess, Linda Freienthal, Lukas Galke Poech, Mike Zhang, Nicolas Legrand, Vladimir Salnikov, Yevhen Kostiuk, Zafar Hussain, Sagandeep Kaur, Agnes Toftg{\aa}rd, Marie Mattson, Kristoffer Nielbo

Multilingual GSM-Symbolic: What determines capability transfer across languages?

Read the original on arXiv AI →

The paper introduces Multilingual GSM‑Symbolic, a large multilingual math dataset with 30,000 question‑answer pairs across 15 languages, designed to evaluate cross‑lingual capability transfer using symbolic templates that generate millions of variations. It quantifies key determinants of transfer—model size, language resource level, reasoning ability, and typological distance—showing that larger models and better reasoning reduce performance gaps between low‑ and high‑resource languages, while typological distance has limited impact. The authors present a predictive framework that explains 92 % of language‑to‑language variation and can estimate unseen language performance within 6 pp, improving to 4.2 pp with only ten target‑language templates.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Machine Learning
Aug 5

M-GATE: Multilingual Grammar, Accuracy in Translation, and Efficiency Benchmark for Large Language Models

arXiv:2608. 03803v1 Announce Type: cross Abstract: Multilingual language models are deployed across a hundred or more languages, yet most benchmarks test whether a model can perform a task _in_ a language rather than whether it commands the language itself, conflating fluency with proficiency.

By Tom\'a\v{s} Burkert, Angelika Peljak-{\L}api\'nska, David Zelen\'y
llmsnlpbenchmarkssafety
More like this →
arXiv AI
Sep 2

The Importance of Being Statistically Earnest: A Critical Re-evaluation of GSM-Symbolic

The paper re‑examines the GSM‑Symbolic benchmark, arguing that its claim of widespread reasoning failures in 25 LLMs is based on weak statistics. Using bootstrapped Generalised Linear Mixed Models on 20 open‑weight models, only eight show significant performance changes, and a systematic shift toward larger integers in the dataset explains many of these effects. The authors also uncover model‑specific failure modes such as variable binding fragility, arithmetic limits, and dual‑task interference, cautioning against blanket conclusions about LLM reasoning.

By Dominika Agnieszka D{\l}ugosz, Arlindo Oliveira, Natalia D\'iaz-Rodr\'iguez
llmsbenchmarks
More like this →
arXiv Computation and Language
Oct 9

Language-Specific Effects of Tokenizer Choice in Multilingual Language Models

The study investigates how the choice of tokenizer influences multilingual language models across 54 tokenizers and 123 models. It finds that tokenizer impact is greater for languages with less training data, and that excluding a language from tokenizer training consistently worsens its performance. While reallocating tokenizer training data to lower-resource languages can help, it does not guarantee improvement, and the best tokenizer properties vary by language, enabling predictive screening of tokenizer candidates.

By Clara Meister, G\"ul Sena Alt{\i}nta\c{s}, Antoine Bosselut
llmsnlp
More like this →