The Flow has not summarised this story yet — read it at arXiv Computation and Language.
NaijaFacts is a multilingual benchmark that tests language models on factual and statistical questions about Nigeria, presented in English, Nigerian Pidgin, Yoruba, Igbo, and Hausa. The study shows that models perform much better on UK and US facts (49–50% accuracy) than on Nigerian facts (20% accuracy), indicating a strong knowledge gap. Language effects are smaller, with performance dropping from English to Yoruba, but overall scale does not improve accuracy on Nigerian statistics.
Multilingual benchmarks for African languages mostly translate generic questions into local languages, measuring linguistic competence but not whether models know facts about Africa. We introduce \nai...
arXiv:2610.14476v1 Announce Type: cross Abstract: Comparing large language models can reveal shared representational structure and systematic differences between them. However, aggregate similarity m...
arXiv:2606. 03219v1 Announce Type: cross Abstract: African languages have very little labelled data, and it is unclear if augmenting the quantity of annotation data reliably enhances downstream performance.
The paper evaluates four metrics—CKA, ANC, GMM dominance per token, and ILO—used to measure cross‑lingual representation sharing in multilingual language models. Across 21 models ranging from 125 M to 14 B parameters, the metrics disagree, and the authors attribute this to anisotropy, where representations cluster in a narrow embedding cone. Only ILO shows a strong, robust correlation with cross‑lingual transfer performance (Spearman’s ρ = 0.90) after controlling for model size, family, and task variation, leading the authors to recommend ILO as the primary metric alongside anisotropy diagnostics.
arXiv:2606. 14347v1 Announce Type: new Abstract: Large language models exhibit strong multilingual capabilities, however, their internal representations are difficult to interpret.