PolyRoute is a benchmark for typed-decision routing that includes 322 native-language messages across 12 languages and 9 writing systems, 156 code-mixed or romanised messages, and 967 optional native training rows. The dataset is annotated by LLMs (Qwen3.8-27B and gemma4-31b) and evaluates a three-way department-routing decision protocol. Leaderboard results show multilingual models outperform English-only baselines, with the best fine-tuned system achieving 96.6% accuracy.
By Mohammed Nayeem Khan, Sarun Kumar K S
The paper evaluates a 307‑million‑parameter multilingual cross‑encoder, Ru‑Jev, on Russian‑language tasks, comparing its original and an augmented release that presents routing labels in multiple surface forms. Experiments show that the augmentation improves accuracy on unfamiliar ordinary words, borrowed jargon, and office slang—up to 0.24 points—while providing no benefit for abbreviations or opaque codes. The study also highlights evaluation hazards and notes that increased accuracy often coincides with reduced risk‑coverage, especially with many candidate labels.
By Vladimir Beskorovainyi
SyntaxBench is a diagnostic benchmark and statistical evaluation framework for character‑level reasoning in large language models, comprising five core tasks—character counting, letter containment, palindrome detection, edit distance, and longest‑string selection—and a harder substring‑extraction stress test called index_to_span. The benchmark uses paired English and random‑string inputs, zero‑, one‑, and four‑shot prompts, and evaluates models from 2B to 32B parameters across multiple reasoning modes. It reports a wide range of metrics, including exact‑match and relaxed accuracy, Cohen’s kappa, McNemar tests, bootstrap confidence intervals, Kendall’s tau, class‑conditional metrics, tokenization analysis, and multiple‑comparison‑corrected tests.
By Mohsen Larni (Department of Computer Science, University of Nevada, Las Vegas), Sobhan Ebrahimi Azar (Department of Computer Science, University of Nevada, Las Vegas), Pouyan Nahed (Department of Computer Science, University of Nevada, Las Vegas), Kazem Taghva (Department of Computer Science, University of Nevada, Las Vegas)
arXiv:2604. 04532v2 Announce Type: replace-cross Abstract: Evaluation language is typically treated as a fixed English default in agentic code benchmarks, yet we show that changing the judge's language can invert backbone rankings.
By Alhasan Mahmood, Samir Abdaljalil, Hasan Kurban
arXiv:2609.15079v1 Announce Type: cross
Abstract: Multi-agent LLM architectures, such as LangChain and AutoGen, largely assume English as the lingua franca for internal inter-agent communication, eve...
By Kushagra Agrawal, Yuming Feng, Man-Fai Leung
The study investigates whether the language of a user’s instruction affects the safety verdicts of agent guards. By translating 1,220 labeled steps from an English benchmark into German, French, Turkish, Swahili, and Amharic while keeping all other data identical, the authors find that many open, small, and purpose‑built guards alter their verdicts when the instruction is translated, with larger effects for rarer languages. The impact varies with guard size, with smaller models showing greater loss of recall and larger models (e.g., Gemini 3.8 Flash) remaining largely unchanged.
By Hasan Demirkiran, Jens Ernstberger