We evaluate large language models (LLMs) as language agents playing goal-directed dialogue games in self-play across 30 languages: the 24 official EU languages plus six others. Unlike static or preference-based evaluation, this paradigm is multi-turn, reference-free and programmatically scored, and because the game mechanics are language-agnostic it extends to a new language by localising a fixed set of prompt and word-list files.
arXiv:2609.39001v1 Announce Type: cross Abstract: LLM services are billed per token and context windows are measured in tokens, yet the number of tokens needed for the same content varies across lang...
arXiv:2609.15079v1 Announce Type: cross Abstract: Multi-agent LLM architectures, such as LangChain and AutoGen, largely assume English as the lingua franca for internal inter-agent communication, eve...
The paper demonstrates that large language model (LLM) evaluators, whether reward‑model based or prompted LLM‑as‑a‑Judge, exhibit significant language bias in multilingual settings. Experiments with semantically identical instruction‑response pairs across 23 languages reveal that lower‑resource languages receive higher scores, a bias that persists across eight open‑weight evaluators and is not detectable by standard pairwise accuracy metrics. The authors link the bias to model uncertainty and language identity, showing it cannot be explained by content difficulty alone.
arXiv:2604. 04532v2 Announce Type: replace-cross Abstract: Evaluation language is typically treated as a fixed English default in agentic code benchmarks, yet we show that changing the judge's language can invert backbone rankings.
The paper investigates whether large language models (LLMs) exhibit language‑specific skill differences by having two identical model instances play a text‑based game in different languages. Using a multilingual extension of TextArena, the authors evaluate three open‑weight models across eight languages and six games, finding that the same model can show markedly different performance—varying win–loss margins, invalid actions, and strategic choices—depending on the language interface. Analyses pinpoint language‑specific failures in spatial reasoning, card‑conditioned decisions, and optimal move selection, and demonstrate that adjusting the intermediate reasoning language can recover much of the lost performance.
Large language models access knowledge inconsistently across languages, but to what extent do they differ in their skill sets when interacting with different languages? This work quantifies cross-ling...
arXiv:2608.28641v1 Announce Type: cross Abstract: Most evaluations for coding agents are conducted exclusively in English, which does not reflect real-world multilingual deployment. We present Termin...
arXiv:2606. 28715v1 Announce Type: cross Abstract: While AI development and evaluation for Southeast Asia (SEA) has grown rapidly, agent capabilities in regional languages are still poorly understood despite its importance to sovereign AI.
arXiv:2606.28715v2 Announce Type: replace-cross Abstract: While AI development and evaluation for Southeast Asia (SEA) has grown rapidly, agent capabilities in regional languages are still poorly und...
arXiv:2609.35820v1 Announce Type: cross Abstract: English-only benchmarks expose only a narrow slice of voice-agent behavior. We introduce $\tau$-Multilingual, extending $\tau$-Voice to Spanish, Braz...
Despite the growing use of LLM-powered agents to solve multi-step tasks in complex environments, existing benchmarks rarely test state preservation, performance across languages, and application to re...
arXiv:2601. 04387v2 Announce Type: replace Abstract: Negotiation is a core component of social intelligence, requiring agents to balance strategic reasoning, cooperation, and social norms.