arXiv AI
AtlasNLP: A Country-Aware Atlas of Dataset Representation in NLP
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
Hugging Face Trending Papers
Aug 31
AtlasNLP: A Country-Aware Atlas of Dataset Representation in NLP
Understanding which countries are represented in NLP datasets is essential for identifying gaps, targeting data collection, measuring progress, and informing AI policy. However, geographic metadata is...
Hugging Face Trending Papers
Oct 27
The State and Fate of Multilingual, Contextual Evaluation in the NLP World
The paper audits 51 multilingual evaluation benchmarks covering 242 datasets and 219 languages, focusing on coverage, representativeness, and rigor. It finds that while coverage is broad, many languages appear in only one benchmark and entire regions are underrepresented, with low-resource languages evaluated on far fewer tasks than high-resource ones. The study also highlights that most datasets rely on English translations, creating artifacts and limiting cultural relevance, and that benchmark contamination through translation leakage and corpus overlap undermines rigor. Recommendations call for natively constructed, culturally grounded, contamination-aware evaluation frameworks that truly serve the communities they aim to assess.
arXiv Computation and Language
Oct 28
The State and Fate of Multilingual, Contextual Evaluation in the NLP World
The paper audits 51 recent multilingual evaluation benchmarks covering 242 datasets and 219 languages, focusing on coverage, representativeness, and rigor. It finds that while coverage is broad, many languages appear in only one benchmark and entire regions are underrepresented; low‑resource languages face a task equity gap. Representativeness is compromised by heavy reliance on English translations, and rigor is weakened by benchmark contamination and parallel‑corpus overlap. The authors propose concrete recommendations for building evaluation frameworks that are natively constructed, culturally grounded, contamination‑aware, and community‑aligned.
arXiv AI
Oct 12
GeoFitBench-LLM: A Benchmark for Assessing Large Language Models' Capabilities in Geospatial Data Requirements Elicitation for Geo-Analytic Tasks
GeoFitBench-LLM is a benchmark designed to evaluate large language models (LLMs) on their ability to elicit geospatial data requirements for geo‑analytic tasks. It focuses on two reasoning dimensions—spatio‑temporal relational and semantic grounding—and uses a provider‑independent schema to encode ground‑truth requirements. Experiments on seven models show that while many can identify necessary resources, accurately specifying fitness‑for‑use remains challenging, with GPT‑5.4 Thinking recovering all resources in 66.67% of cases.
arXiv AI
Sep 25
From Policy Documents to Structured Survey Responses: Evaluating Large Language Models for Policy Monitoring
The paper explores using large language models (LLMs) as AI respondents to convert policy documents into structured survey responses. It introduces a long-context in‑context learning pipeline that maps policy text to predefined survey categories such as policy instruments, target groups, and thematic areas, and includes a secondary LLM validation step. Evaluation on a multi‑country dataset shows high agreement (84‑95%) with human responses for structured indicators, though free‑text fields differ, indicating that hybrid human‑AI workflows can enhance policy monitoring efficiency while still requiring human oversight.
arXiv AI
Sep 12
Geospatial AI, Dataverse Metadata, and the Study of Place-Based Government
The paper presents a knowledge graph built from Harvard Dataverse’s public data, linking 102,650 datasets to 215,985 nodes and 528,003 edges that include keywords, publications, subjects, journals, and locations. About 43,991 datasets contain geospatial metadata, and 7,654 are identified as policy‑relevant, with elections and legislatures forming the largest cluster. The authors highlight the challenge of place resolution—disconnected nodes representing the same location—and propose the graph as a testbed for AI‑driven metadata enrichment and entity resolution, noting a bias toward American city‑level data.