Hugging Face Trending Papers

PolyRoute: A Twelve-Language and Code-Mixed Benchmark for Typed-Decision Routing, with Baselines and a Leaderboard

Read the original on Hugging Face Trending Papers →

PolyRoute is a benchmark that evaluates typed‑decision routing models across 12 languages and code‑mixed text, providing 322 native‑language messages, 156 code‑mixed or romanised messages, and 967 optional native training rows. The dataset includes a three‑way routing protocol with labels generated by Qwen3.8‑27B and verified by gemma4‑31b, achieving high agreement (97.2%) and low character‑level similarity to training data. A leaderboard of 13 systems shows performance ranging from 61.2% for an English‑root checkpoint to 96.6% for the best fine‑tuned multilingual model, with TF‑IDF logistic regression achieving 91.9%. whyItMatters":"The benchmark offers a standardized, multilingual yardstick for assessing typed‑decision routing models, enabling direct comparison across languages and code‑mixed contexts."

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at Hugging Face Trending Papers.

arXiv AI
6d ago

PolyRoute: A Twelve-Language and Code-Mixed Benchmark for Typed-Decision Routing, with Baselines and a Leaderboard

PolyRoute is a benchmark for typed-decision routing that includes 322 native-language messages across 12 languages and 9 writing systems, 156 code-mixed or romanised messages, and 967 optional native training rows. The dataset is annotated by LLMs (Qwen3.8-27B and gemma4-31b) and evaluates a three-way department-routing decision protocol. Leaderboard results show multilingual models outperform English-only baselines, with the best fine-tuned system achieving 96.6% accuracy.

By Mohammed Nayeem Khan, Sarun Kumar K S
arXiv AI
Nov 6

Label-Wording Augmentation in a Typed-Decision Encoder: Gains on Unfamiliar Words, Not on Abbreviations or Codes

The paper evaluates a 307‑million‑parameter multilingual cross‑encoder, Ru‑Jev, on Russian‑language tasks, comparing its original and an augmented release that presents routing labels in multiple surface forms. Experiments show that the augmentation improves accuracy on unfamiliar ordinary words, borrowed jargon, and office slang—up to 0.24 points—while providing no benefit for abbreviations or opaque codes. The study also highlights evaluation hazards and notes that increased accuracy often coincides with reduced risk‑coverage, especially with many candidate labels.

By Vladimir Beskorovainyi
arXiv AI
Oct 5

SyntaxBench: A Statistical Diagnostic Framework for Character-Level Reasoning in Large Language Models

SyntaxBench is a diagnostic benchmark and statistical evaluation framework for character‑level reasoning in large language models, comprising five core tasks—character counting, letter containment, palindrome detection, edit distance, and longest‑string selection—and a harder substring‑extraction stress test called index_to_span. The benchmark uses paired English and random‑string inputs, zero‑, one‑, and four‑shot prompts, and evaluates models from 2B to 32B parameters across multiple reasoning modes. It reports a wide range of metrics, including exact‑match and relaxed accuracy, Cohen’s kappa, McNemar tests, bootstrap confidence intervals, Kendall’s tau, class‑conditional metrics, tokenization analysis, and multiple‑comparison‑corrected tests.

By Mohsen Larni (Department of Computer Science, University of Nevada, Las Vegas), Sobhan Ebrahimi Azar (Department of Computer Science, University of Nevada, Las Vegas), Pouyan Nahed (Department of Computer Science, University of Nevada, Las Vegas), Kazem Taghva (Department of Computer Science, University of Nevada, Las Vegas)
arXiv AI
Oct 16

Language Is a Security Surface for Agent Guards

The study investigates whether the language of a user’s instruction affects the safety verdicts of agent guards. By translating 1,220 labeled steps from an English benchmark into German, French, Turkish, Swahili, and Amharic while keeping all other data identical, the authors find that many open, small, and purpose‑built guards alter their verdicts when the instruction is translated, with larger effects for rarer languages. The impact varies with guard size, with smaller models showing greater loss of recall and larger models (e.g., Gemini 3.8 Flash) remaining largely unchanged.

By Hasan Demirkiran, Jens Ernstberger