The paper investigates whether large language models (LLMs) exhibit language‑specific skill differences by having two identical model instances play a text‑based game in different languages. Using a multilingual extension of TextArena, the authors evaluate three open‑weight models across eight languages and six games, finding that the same model can show markedly different performance—varying win–loss margins, invalid actions, and strategic choices—depending on the language interface. Analyses pinpoint language‑specific failures in spatial reasoning, card‑conditioned decisions, and optimal move selection, and demonstrate that adjusting the intermediate reasoning language can recover much of the lost performance.
Large language models access knowledge inconsistently across languages, but to what extent do they differ in their skill sets when interacting with different languages? This work quantifies cross-ling...
arXiv:2609.37104v1 Announce Type: cross Abstract: Open-source reasoning models provide unequal access to reasoning capability across languages. When a model can solve a problem but cannot deliver a c...
The paper demonstrates that large language model (LLM) evaluators, whether reward‑model based or prompted LLM‑as‑a‑Judge, exhibit significant language bias in multilingual settings. Experiments with semantically identical instruction‑response pairs across 23 languages reveal that lower‑resource languages receive higher scores, a bias that persists across eight open‑weight evaluators and is not detectable by standard pairwise accuracy metrics. The authors link the bias to model uncertainty and language identity, showing it cannot be explained by content difficulty alone.
arXiv:2608. 15964v1 Announce Type: cross Abstract: Language-specific competency (LSC) is the phenomenon of a language model performing better or worse depending on the language of the prompt.
arXiv:2608. 08447v1 Announce Type: cross Abstract: Multilingual reasoning models are commonly evaluated by whether they arrive at the correct answer, but not by whether they preserve the intended language while reasoning and responding.
M‑SQE is a post‑retrieval framework that estimates the quality of multilingual agent skills by combining a Theory view (intrinsic quality) and an Action view (task‑grounded utility) into a domain‑conditioned score. It was evaluated on general, tool‑use, and cultural skill‑use domains, showing a task‑success improvement of at least +3.5 points over baselines across three retrievers. The method notably boosts performance for low‑resource languages, raising Hindi by +12.9 pp and Swahili by +5.6 pp, and achieves strong results across six cultural regions, advancing linguistic and cultural equality in agentic skill use.
arXiv:2610.18354v1 Announce Type: new Abstract: Game-theoretic benchmarks claim that large language model (LLM) strategic decision-making varies across game types, interaction structures, and capabil...
The study explores how the language used for reasoning affects retrieval‑augmented generation (RAG) in a monolingual German setting. Using a German RAG question‑answering testbed based on the tabletop game The Dark Eye, the authors show that aligning the reasoning language with the query and retrieved documents improves performance, with German reasoning outperforming French reasoning. However, German reasoning still does not surpass the model’s native English reasoning, indicating that native multilingual reasoning is necessary for optimal results.
When a tool-using agent is given the same task in a different language, does it still take the same steps? Multilingual evaluation rarely asks: it compares final answers and discards the actions.
arXiv:2601. 19082v2 Announce Type: replace Abstract: Large language models (LLMs) are increasingly deployed as autonomous agents that negotiate, coordinate, and act on behalf of users.
The paper introduces CodeHack, a library of code-based skills with natural-language descriptions designed to improve language agents in complex environments like NetHack. By allowing agents to invoke reusable skills instead of selecting individual actions, the study shows that skill-based agents nearly triple game progression and cut inference cost by 86% in zero‑shot settings, while still retaining the option to fall back on primitive actions. In reinforcement learning, skill-based agents learn faster, achieving a 7.2× larger average gain in dungeon level within the same training budget.
SpireBench evaluates long‑horizon strategic play by having large language models play the full commercial deck‑builder Slay the Spire, a game where early choices determine long‑term success. Across 20 LLMs at the highest difficulty, frontier models beat heuristic baselines and match average human performance but lag far behind experts, especially as runs lengthen. The benchmark reveals that strong models excel in early fights but misjudge risk over a run, and that their performance depends partly on prior game knowledge rather than on memories written during play.