arXiv AI

PolyRoute: A Twelve-Language and Code-Mixed Benchmark for Typed-Decision Routing, with Baselines and a Leaderboard

PolyRoute is a benchmark for typed-decision routing that includes 322 native-language messages across 12 languages and 9 writing systems, 156 code-mixed or romanised messages, and 967 optional native training rows. The dataset is annotated by LLMs (Qwen3.8-27B and gemma4-31b) and evaluates a three-way department-routing decision protocol. Leaderboard results show multilingual models outperform English-only baselines, with the best fine-tuned system achieving 96.6% accuracy.

Hugging Face Trending Papers
6d ago

PolyRoute: A Twelve-Language and Code-Mixed Benchmark for Typed-Decision Routing, with Baselines and a Leaderboard

PolyRoute is a benchmark that evaluates typed‑decision routing models across 12 languages and code‑mixed text, providing 322 native‑language messages, 156 code‑mixed or romanised messages, and 967 optional native training rows. The dataset includes a three‑way routing protocol with labels generated by Qwen3.8‑27B and verified by gemma4‑31b, achieving high agreement (97.2%) and low character‑level similarity to training data. A leaderboard of 13 systems shows performance ranging from 61.2% for an English‑root checkpoint to 96.6% for the best fine‑tuned multilingual model, with TF‑IDF logistic regression achieving 91.9%. whyItMatters":"The benchmark offers a standardized, multilingual yardstick for assessing typed‑decision routing models, enabling direct comparison across languages and code‑mixed contexts."

arXiv AI
Nov 6

Label-Wording Augmentation in a Typed-Decision Encoder: Gains on Unfamiliar Words, Not on Abbreviations or Codes

The paper evaluates a 307‑million‑parameter multilingual cross‑encoder, Ru‑Jev, on Russian‑language tasks, comparing its original and an augmented release that presents routing labels in multiple surface forms. Experiments show that the augmentation improves accuracy on unfamiliar ordinary words, borrowed jargon, and office slang—up to 0.24 points—while providing no benefit for abbreviations or opaque codes. The study also highlights evaluation hazards and notes that increased accuracy often coincides with reduced risk‑coverage, especially with many candidate labels.

By Vladimir Beskorovainyi
arXiv AI
Oct 16

Language Is a Security Surface for Agent Guards

The study investigates whether the language of a user’s instruction affects the safety verdicts of agent guards. By translating 1,220 labeled steps from an English benchmark into German, French, Turkish, Swahili, and Amharic while keeping all other data identical, the authors find that many open, small, and purpose‑built guards alter their verdicts when the instruction is translated, with larger effects for rarer languages. The impact varies with guard size, with smaller models showing greater loss of recall and larger models (e.g., Gemini 3.8 Flash) remaining largely unchanged.

By Hasan Demirkiran, Jens Ernstberger
arXiv AI
Oct 31

One Confidence Threshold, Unequal Coverage: Language Gaps in Typed Decision Interfaces and How Translating Inputs Reduces Them

The study evaluates whether a single confidence threshold in typed decision interfaces treats all languages equally. Using 1,000 items in 51 languages across three vendors, it finds that English consistently achieves higher coverage—clearing the 0.9 threshold 8.6–11.7 percentage points more often than other languages—despite similar accuracy. Even when thresholds are re‑weighted or calibrated per language, the gap persists, and translating inputs into English reduces but does not eliminate it. The authors recommend reporting per‑language coverage at the operating threshold.

By Junwei Yao, Yongxi Zhou, Zike Yang, Zhijun Wang, Jianan Liu, Yanxian Chen, Wenbo Ye
arXiv Computation and Language
Oct 19

Who Pays for Efficient Inference? Confidence-Based Small-to-Large LLM Routing Across Tasks and Languages

The paper evaluates confidence‑based routing of small‑to‑large language models across multiple tasks, languages, and model families, covering 72 configurations and over 42,000 predictions. It finds that confidence routing improves performance on reading comprehension and topic classification datasets, but offers little benefit on a knowledge‑reasoning benchmark where small models remain confidently wrong. The study also shows that global confidence routing can create large language‑specific performance gaps, which equal‑quota routing mitigates with minimal overall accuracy loss.

By Md Sabbir Hossen, Anichur Rahman, Pabon Shaha, Airin Afroj Aishi, Andrew H. Sung, Md Shohel Rana
arXiv AI
Oct 5

SyntaxBench: A Statistical Diagnostic Framework for Character-Level Reasoning in Large Language Models

SyntaxBench is a diagnostic benchmark and statistical evaluation framework for character‑level reasoning in large language models, comprising five core tasks—character counting, letter containment, palindrome detection, edit distance, and longest‑string selection—and a harder substring‑extraction stress test called index_to_span. The benchmark uses paired English and random‑string inputs, zero‑, one‑, and four‑shot prompts, and evaluates models from 2B to 32B parameters across multiple reasoning modes. It reports a wide range of metrics, including exact‑match and relaxed accuracy, Cohen’s kappa, McNemar tests, bootstrap confidence intervals, Kendall’s tau, class‑conditional metrics, tokenization analysis, and multiple‑comparison‑corrected tests.

By Mohsen Larni (Department of Computer Science, University of Nevada, Las Vegas), Sobhan Ebrahimi Azar (Department of Computer Science, University of Nevada, Las Vegas), Pouyan Nahed (Department of Computer Science, University of Nevada, Las Vegas), Kazem Taghva (Department of Computer Science, University of Nevada, Las Vegas)
arXiv Machine Learning
Aug 5

M-GATE: Multilingual Grammar, Accuracy in Translation, and Efficiency Benchmark for Large Language Models

arXiv:2608. 03803v1 Announce Type: cross Abstract: Multilingual language models are deployed across a hundred or more languages, yet most benchmarks test whether a model can perform a task _in_ a language rather than whether it commands the language itself, conflating fluency with proficiency.

By Tom\'a\v{s} Burkert, Angelika Peljak-{\L}api\'nska, David Zelen\'y
arXiv AI
3d ago

Evaluating Jev on Arabic: Comparing Decision Models and LLMs Across Arabic Tasks and Dialects

The paper evaluates the Jev system on Arabic, comparing it with 66 other models across 43 tasks and 9 task families. Jev 1.14.0 scores a mean task-primary metric of 69.24, slightly below Gemma-4-31B’s 71.81 but above Jev 1.13.0’s 68.77. The study also examines performance differences between Modern Standard Arabic and dialects, noting variable gaps and a notable Palestinian banking penalty for Jev 1.14.0.

By Hossam Elsafty, Tobias Deu{\ss}er, Lorenz Sparrenberg, Rajkumar Ramamurthy, Rafet Sifa