Hugging Face Trending Papers
The State and Fate of Multilingual, Contextual Evaluation in the NLP World
Read the original on Hugging Face Trending Papers →The Flow has not summarised this story yet — read it at Hugging Face Trending Papers.
Related stories
arXiv Computation and Language
1d ago
The State and Fate of Multilingual, Contextual Evaluation in the NLP World
The paper audits 51 recent multilingual evaluation benchmarks covering 242 datasets and 219 languages, focusing on coverage, representativeness, and rigor. It finds that while coverage is broad, many languages appear in only one benchmark and entire regions are underrepresented; low‑resource languages face a task equity gap. Representativeness is compromised by heavy reliance on English translations, and rigor is weakened by benchmark contamination and parallel‑corpus overlap. The authors propose concrete recommendations for building evaluation frameworks that are natively constructed, culturally grounded, contamination‑aware, and community‑aligned.
arXiv Machine Learning
Aug 17
Language-Specific Gaps in AI Safety Training Datasets
arXiv:2608. 13695v1 Announce Type: cross Abstract: Large language model providers routinely cite multilingual safety benchmarks spanning a dozen or more languages as evidence that their models are safe for non-English-speaking users.
arXiv Computation and Language
Sep 22
BabelArena: A Large-Scale Multilingual Benchmark for LLM Agents
arXiv:2609.23490v1 Announce Type: new Abstract: Large language model (LLM) agents increasingly execute multi-step workflows through tool use and interaction with users and environments. However, curr...
arXiv Computation and Language
Sep 23
FineWeb-CLaR: Culture, Language, and Region Annotations for Benchmark-Aligned Corpus Auditing
arXiv:2609.25298v1 Announce Type: new Abstract: Cultural evaluation coverage and robustness in language models are difficult to diagnose because pretraining corpora and cultural benchmarks are rarely...
arXiv Computation and Language
Aug 27
Apples to Apples? Towards Comparable Crosslingual Language Model Evaluation
The paper investigates how to fairly compare language models across languages, noting that current evaluation methods vary widely and lack empirical validation. By training controlled monolingual models on parallel data and testing multilingual LLMs, the authors find that many normalized metrics suffer from biases due to tokenization, encoding, and orthographic differences. Instead, they recommend using sentence‑level negative log‑likelihood over semantically equivalent sequences for more reliable cross‑lingual comparisons.
arXiv AI
Jul 23
On the Systematic Challenges of Culturally Loaded Machine Translation: Dream of the Red Chamber as the Cultural Lens
arXiv:2607. 20241v1 Announce Type: cross Abstract: Culturally loaded translation poses unique challenges for machine translation (MT), as meanings are deeply embedded in socio-cultural contexts beyond surface linguistic forms.