NaijaFacts is a multilingual benchmark that tests language models on factual and statistical questions about Nigeria, presented in English, Nigerian Pidgin, Yoruba, Igbo, and Hausa. The study shows that models perform much better on UK and US facts (49–50% accuracy) than on Nigerian facts (20% accuracy), indicating a strong knowledge gap. Language effects are smaller, with performance dropping from English to Yoruba, but overall scale does not improve accuracy on Nigerian statistics.
Multilingual benchmarks for African languages mostly translate generic questions into local languages, measuring linguistic competence but not whether models know facts about Africa. We introduce \nai...
arXiv:2609.36205v1 Announce Type: new Abstract: We study how Gemma 4 31B represents African languages and responds to cultural steering. The first study compares nine African languages and three cont...
arXiv:2606. 07422v1 Announce Type: cross Abstract: Large language models are increasingly used to answer culturally grounded questions across languages, yet it remains unclear whether local cultural knowledge is better accessed through English or the local language.
The paper surveys NLP research on Nigeria’s three major low‑resource languages—Hausa, Yoruba, and Igbo—covering over 500 languages spoken by 175 million people. It reviews 293 studies, finding that only 27.6% produced new linguistic resources, indicating a heavy reliance on repurposing existing data. The authors highlight under‑explored challenges such as morphological analysis and diacritic representation, and call for collaborative resource enrichment and community support to advance NaijaNLP and low‑resource NLP more broadly.
The paper introduces TriviaRoomQA, a multilingual benchmark of 3,300 parallel multiple-choice questions across 288 topics, plus an additional 5,340 French-only questions. It evaluates 30 open-weight large language models (7–70 B parameters) from various regions, finding strong performance on knowledge-intensive subjects like history and geography but weaker results on everyday popular‑culture topics such as celebrities, music, movies, and news. The study also shows that model performance varies across languages for the same questions, indicating that factual knowledge access is not always language‑independent.
Quiz rooms, trivia nights, and quiz shows challenge human knowledge across a wide range of topics, from canonical facts to everyday culture. In this paper, we examine whether large language models (LLMs) can perform competitively in such settings, using quiz-style questions to test them on both common and niche topics.
The paper "Where's Waldo? Query-language Preference under Cross-lingual Knowledge Disparities" introduces the Waldo benchmark, a multilingual QA dataset built from Wikipedia that focuses on knowledge gaps and conflicts across languages. It evaluates eight models in five languages and finds that when a fact is missing in one language, models tend to use evidence from the other language, but when conflicting accounts exist, responses align strongly with the query language, leading to different answers for semantically identical questions. The study also explores mitigation strategies, including ablating attention heads and LoRA-based training, which can reduce the preference gap by up to 61.5%.
arXiv:2606. 24460v1 Announce Type: cross Abstract: Commercial large language models bill, scale latency, and budget context per token.
AfrIFact is a new dataset designed to support automatic fact‑checking in ten African languages and English, covering information retrieval, evidence extraction, and fact verification. Experiments show that current embedding models struggle with cross‑lingual retrieval, and that cultural and news documents are easier to retrieve than healthcare documents. Large language models also lack robust multilingual fact‑verification, but few‑shot prompting and task‑specific fine‑tuning can significantly improve performance.
The paper introduces the concept of Language Specific Knowledge (LSK), showing that multilingual language models can answer certain queries better when prompted in a language other than English, sometimes even in low‑resource languages. It defines a language‑selection problem and presents several baseline methods, including the authors’ LSKExtractor, to empirically demonstrate that choosing the optimal language can improve question‑answering performance across datasets covering cultural and social norms. Experiments reveal non‑intuitive mappings, such as Gemma models excelling on Chinese and Middle Eastern topics in Spanish and Qwen models performing best on authority and responsibility queries in Arabic and Chinese.
The study evaluates six state‑of‑the‑art large language models on 400 adversarial prompts in Yoruba and Nigerian Pidgin, categorised into four harm types. Native‑speaker annotators scored responses and assessed reliability with Cohen’s κ, revealing that many models failed to understand the prompts, leading to low rejection rates in the African languages. Harmful content was frequently produced, with attack success rates higher in Nigerian languages and significant cross‑lingual safety gaps, especially for GPT‑4o Mini and Aya 101.
arXiv:2606. 20255v1 Announce Type: cross Abstract: We introduce the Meaning Intelligence Framework (MIF), a nine-dimension annotation and evaluation schema for Nigerian public discourse that separates surface sentiment from true communicative intent.