PolyRoute is a benchmark that evaluates typed‑decision routing models across 12 languages and code‑mixed text, providing 322 native‑language messages, 156 code‑mixed or romanised messages, and 967 optional native training rows. The dataset includes a three‑way routing protocol with labels generated by Qwen3.8‑27B and verified by gemma4‑31b, achieving high agreement (97.2%) and low character‑level similarity to training data. A leaderboard of 13 systems shows performance ranging from 61.2% for an English‑root checkpoint to 96.6% for the best fine‑tuned multilingual model, with TF‑IDF logistic regression achieving 91.9%.
whyItMatters":"The benchmark offers a standardized, multilingual yardstick for assessing typed‑decision routing models, enabling direct comparison across languages and code‑mixed contexts."
The paper evaluates a 307‑million‑parameter multilingual cross‑encoder, Ru‑Jev, on Russian‑language tasks, comparing its original and an augmented release that presents routing labels in multiple surface forms. Experiments show that the augmentation improves accuracy on unfamiliar ordinary words, borrowed jargon, and office slang—up to 0.24 points—while providing no benefit for abbreviations or opaque codes. The study also highlights evaluation hazards and notes that increased accuracy often coincides with reduced risk‑coverage, especially with many candidate labels.
By Vladimir Beskorovainyi
The study investigates whether the language of a user’s instruction affects the safety verdicts of agent guards. By translating 1,220 labeled steps from an English benchmark into German, French, Turkish, Swahili, and Amharic while keeping all other data identical, the authors find that many open, small, and purpose‑built guards alter their verdicts when the instruction is translated, with larger effects for rarer languages. The impact varies with guard size, with smaller models showing greater loss of recall and larger models (e.g., Gemini 3.8 Flash) remaining largely unchanged.
By Hasan Demirkiran, Jens Ernstberger
arXiv:2610.18420v1 Announce Type: cross
Abstract: We trained Agaram 2.5B, a 2.5-billion-parameter Tamil-English language model, from scratch on about 50 billion tokens (58% Tamil, the rest English, m...
By Ramchand Kumaresan, Madhusudhanan G, Kaushik Kezendirane
arXiv:2609.15079v1 Announce Type: cross
Abstract: Multi-agent LLM architectures, such as LangChain and AutoGen, largely assume English as the lingua franca for internal inter-agent communication, eve...
By Kushagra Agrawal, Yuming Feng, Man-Fai Leung
The study evaluates whether a single confidence threshold in typed decision interfaces treats all languages equally. Using 1,000 items in 51 languages across three vendors, it finds that English consistently achieves higher coverage—clearing the 0.9 threshold 8.6–11.7 percentage points more often than other languages—despite similar accuracy. Even when thresholds are re‑weighted or calibrated per language, the gap persists, and translating inputs into English reduces but does not eliminate it. The authors recommend reporting per‑language coverage at the operating threshold.
By Junwei Yao, Yongxi Zhou, Zike Yang, Zhijun Wang, Jianan Liu, Yanxian Chen, Wenbo Ye
The paper evaluates confidence‑based routing of small‑to‑large language models across multiple tasks, languages, and model families, covering 72 configurations and over 42,000 predictions. It finds that confidence routing improves performance on reading comprehension and topic classification datasets, but offers little benefit on a knowledge‑reasoning benchmark where small models remain confidently wrong. The study also shows that global confidence routing can create large language‑specific performance gaps, which equal‑quota routing mitigates with minimal overall accuracy loss.
By Md Sabbir Hossen, Anichur Rahman, Pabon Shaha, Airin Afroj Aishi, Andrew H. Sung, Md Shohel Rana
arXiv:2604. 04532v2 Announce Type: replace-cross Abstract: Evaluation language is typically treated as a fixed English default in agentic code benchmarks, yet we show that changing the judge's language can invert backbone rankings.
By Alhasan Mahmood, Samir Abdaljalil, Hasan Kurban
SyntaxBench is a diagnostic benchmark and statistical evaluation framework for character‑level reasoning in large language models, comprising five core tasks—character counting, letter containment, palindrome detection, edit distance, and longest‑string selection—and a harder substring‑extraction stress test called index_to_span. The benchmark uses paired English and random‑string inputs, zero‑, one‑, and four‑shot prompts, and evaluates models from 2B to 32B parameters across multiple reasoning modes. It reports a wide range of metrics, including exact‑match and relaxed accuracy, Cohen’s kappa, McNemar tests, bootstrap confidence intervals, Kendall’s tau, class‑conditional metrics, tokenization analysis, and multiple‑comparison‑corrected tests.
By Mohsen Larni (Department of Computer Science, University of Nevada, Las Vegas), Sobhan Ebrahimi Azar (Department of Computer Science, University of Nevada, Las Vegas), Pouyan Nahed (Department of Computer Science, University of Nevada, Las Vegas), Kazem Taghva (Department of Computer Science, University of Nevada, Las Vegas)
arXiv:2608.23023v2 Announce Type: new
Abstract: An LLM router picks which model should answer each query. The appeal is that models fail on different questions. Whatever single model is best overall...
By Janghoon Lee (Redrob)
arXiv:2608. 03803v1 Announce Type: cross Abstract: Multilingual language models are deployed across a hundred or more languages, yet most benchmarks test whether a model can perform a task _in_ a language rather than whether it commands the language itself, conflating fluency with proficiency.
By Tom\'a\v{s} Burkert, Angelika Peljak-{\L}api\'nska, David Zelen\'y
The paper evaluates the Jev system on Arabic, comparing it with 66 other models across 43 tasks and 9 task families. Jev 1.14.0 scores a mean task-primary metric of 69.24, slightly below Gemma-4-31B’s 71.81 but above Jev 1.13.0’s 68.77. The study also examines performance differences between Modern Standard Arabic and dialects, noting variable gaps and a notable Palestinian banking penalty for Jev 1.14.0.
By Hossam Elsafty, Tobias Deu{\ss}er, Lorenz Sparrenberg, Rajkumar Ramamurthy, Rafet Sifa