arXiv:2607. 08731v1 Announce Type: cross Abstract: A national language model offers a linguistic community its own instrument for measuring what its citizens say and value.
By Manuel Pita
arXiv:2610.20461v1 Announce Type: new
Abstract: Character Error Rate (CER), the standard metric for OCR and vision-language models (VLMs), charges a typo as much as a wrong name. We introduce semCER,...
By Nguyen Nguyen, Yongyi Zang
Character Error Rate (CER), the standard metric for OCR and vision-language models (VLMs), charges a typo as much as a wrong name. We introduce semCER, which keeps CER's alignment and denominator and...
A national language model offers a linguistic community its own instrument for measuring what its citizens say and value. Portugal's AMALIA, a publicly funded 9B-parameter model for European Portuguese, appears competitive on agreement alone: asked to code the moral foundation of authority, it agrees with trained human coders to within six F1 points of open models eight to thirteen times its size.
The study evaluates whether a single confidence threshold in typed decision interfaces treats all languages equally. Using 1,000 items in 51 languages across three vendors, it finds that English consistently achieves higher coverage—clearing the 0.9 threshold 8.6–11.7 percentage points more often than other languages—despite similar accuracy. Even when thresholds are re‑weighted or calibrated per language, the gap persists, and translating inputs into English reduces but does not eliminate it. The authors recommend reporting per‑language coverage at the operating threshold.
By Junwei Yao, Yongxi Zhou, Zike Yang, Zhijun Wang, Jianan Liu, Yanxian Chen, Wenbo Ye
The study investigates how exposing large language models to Arabic translations of English benchmark tasks can hide data contamination from standard English-only detection methods. By testing four instruction-tuned models with increasing Arabic exposure, the authors find that traditional probes like TS-Guessing and Min-K%++ fail to detect contamination, yet English performance still improves. They propose a new Translation-Aware Contamination Detection (TACD) method that uses cross-lingual consistency and choice reordering to reveal contamination effects that are otherwise invisible to English-only diagnostics.
By Chaymaa Abbas, Nour Shammaa, Mariette Awad