arXiv AI
Aug 10
Geo-Spatial Concept Probing of Large Language Models: Abstraction, Compositionality, and Grounding
arXiv:2608. 07353v1 Announce Type: cross Abstract: Understanding concepts is fundamental to generalization.
GeoBenchLLM: A Comprehensive Benchmark for Evaluating LLMs on Geo-Related Tasks
arXiv:2608. 07411v1 Announce Type: new Abstract: In the context of geodata, existing Large Language Models have often been studied in a homogeneous setting, which has considerably limited insights into their generalization capabilities.
Related stories
arXiv AI
Aug 5
MultiGlobeQA: A Multilingual and Globally Diverse Benchmark for Geospatial Reasoning
arXiv:2608. 03882v1 Announce Type: cross Abstract: Geospatial reasoning, i.
Hugging Face Trending Papers
Aug 3
GISAgentBench: A Practitioner-Sourced Benchmark for Evaluating LLM Agents on GIS Tasks
Geographic Information System (GIS) professionals rely on multi-step spatial analysis workflows to support decision-making in urban planning, disaster response, and environmental monitoring. The process is tedious, time-consuming, and error-prone.
arXiv AI
Jul 8
Data Analysis in the Wild: Benchmarking Large Language Models Against Real-World Data Complexities
arXiv:2607. 06482v1 Announce Type: cross Abstract: Current benchmarks for evaluating Large Language Models (LLMs) in data analysis often fail to reflect real-world settings.
arXiv Computer Vision
Aug 27
GTPred: Benchmarking MLLMs for Interpretable Geo-localization and Time-of-capture Prediction
GTPred is a new benchmark for geo‑temporal prediction that evaluates multi‑modal large language models (MLLMs) on 370 images taken across 120 years worldwide. It assesses predictions by matching both the year and a hierarchical location sequence, and includes annotated reasoning chains to test intermediate reasoning. Experiments on 15 MLLMs show that while visual perception is strong, models still lack world knowledge and geo‑temporal reasoning, and that adding temporal data improves location inference.
Hugging Face Trending Papers
Jul 29
OVEarth-Bench: Evaluating Category Breadth and Query Diversity for Open-Vocabulary Earth Observation
Open-vocabulary Earth observation (EO) aims to localize geospatial concepts specified in natural language rather than a fixed label set. Existing benchmarks, however, usually cover narrow category vocabularies or limited query forms.
arXiv AI
Jun 10
LakeQA: An Exploratory QA Benchmark over a Million-Scale Data Lake
arXiv:2606. 10460v1 Announce Type: cross Abstract: Recent large language models (LLMs) have shown rapid progress in reading-based question answering (QA), where evidence is explicitly provided or can be trivially retrieved.
arXiv Computation and Language
Sep 14
Zipbench: Low-Cost Framework for Compressing Comprehensive Benchmarks of Large Language Models
arXiv:2609.12475v1 Announce Type: new Abstract: Comprehensive benchmark suites are essential for improving large language models (LLMs), but many widely used benchmarks are redundant, making evaluati...
arXiv AI
Jun 2
LLM-WikiRace Benchmark: How Far Can LLMs Plan over Real-World Knowledge Graphs?
arXiv:2602. 16902v4 Announce Type: replace Abstract: We introduce LLM-Wikirace, a benchmark for evaluating planning, reasoning, and world knowledge in large language models (LLMs).
arXiv Machine Learning
Jul 3
Bringing Agentic Search to Earth Observation Data Discovery
arXiv:2607. 02387v1 Announce Type: cross Abstract: NASA and its data centers hold thousands of geoscience datasets and tools like Worldview, Giovanni, the Science Discovery Engine, and Harmony.
Hugging Face Trending Papers
Jun 22
Graph-Enhanced Large Language Models for Spatial Search
There have been many recent improvements in the ability of Large Language Models (LLMs) to perform complex tasks and answer domain-specific questions through techniques like Retrieval Augmented Generation (RAG). However, reasoning abilities of LLMs, including spatial reasoning abilities, are still lacking.
arXiv AI
Sep 1
AtlasNLP: A Country-Aware Atlas of Dataset Representation in NLP
arXiv:2608.30107v1 Announce Type: cross Abstract: Understanding which countries are represented in NLP datasets is essential for identifying gaps, targeting data collection, measuring progress, and i...