arXiv AI
Aug 10
Geo-Spatial Concept Probing of Large Language Models: Abstraction, Compositionality, and Grounding
arXiv:2608. 07353v1 Announce Type: cross Abstract: Understanding concepts is fundamental to generalization.
GeoBenchLLM: A Comprehensive Benchmark for Evaluating LLMs on Geo-Related Tasks
arXiv:2608. 07411v1 Announce Type: new Abstract: In the context of geodata, existing Large Language Models have often been studied in a homogeneous setting, which has considerably limited insights into their generalization capabilities.
Related stories
arXiv AI
Aug 5
MultiGlobeQA: A Multilingual and Globally Diverse Benchmark for Geospatial Reasoning
arXiv:2608. 03882v1 Announce Type: cross Abstract: Geospatial reasoning, i.
Hugging Face Trending Papers
Aug 3
GISAgentBench: A Practitioner-Sourced Benchmark for Evaluating LLM Agents on GIS Tasks
Geographic Information System (GIS) professionals rely on multi-step spatial analysis workflows to support decision-making in urban planning, disaster response, and environmental monitoring. The process is tedious, time-consuming, and error-prone.
arXiv AI
Jul 8
Data Analysis in the Wild: Benchmarking Large Language Models Against Real-World Data Complexities
arXiv:2607. 06482v1 Announce Type: cross Abstract: Current benchmarks for evaluating Large Language Models (LLMs) in data analysis often fail to reflect real-world settings.
Hugging Face Trending Papers
Jul 29
OVEarth-Bench: Evaluating Category Breadth and Query Diversity for Open-Vocabulary Earth Observation
Open-vocabulary Earth observation (EO) aims to localize geospatial concepts specified in natural language rather than a fixed label set. Existing benchmarks, however, usually cover narrow category vocabularies or limited query forms.
arXiv AI
Jun 10
LakeQA: An Exploratory QA Benchmark over a Million-Scale Data Lake
arXiv:2606. 10460v1 Announce Type: cross Abstract: Recent large language models (LLMs) have shown rapid progress in reading-based question answering (QA), where evidence is explicitly provided or can be trivially retrieved.
arXiv AI
Jun 2
LLM-WikiRace Benchmark: How Far Can LLMs Plan over Real-World Knowledge Graphs?
arXiv:2602. 16902v4 Announce Type: replace Abstract: We introduce LLM-Wikirace, a benchmark for evaluating planning, reasoning, and world knowledge in large language models (LLMs).
arXiv Machine Learning
Jul 3
Bringing Agentic Search to Earth Observation Data Discovery
arXiv:2607. 02387v1 Announce Type: cross Abstract: NASA and its data centers hold thousands of geoscience datasets and tools like Worldview, Giovanni, the Science Discovery Engine, and Harmony.
Hugging Face Trending Papers
Jun 22
Graph-Enhanced Large Language Models for Spatial Search
There have been many recent improvements in the ability of Large Language Models (LLMs) to perform complex tasks and answer domain-specific questions through techniques like Retrieval Augmented Generation (RAG). However, reasoning abilities of LLMs, including spatial reasoning abilities, are still lacking.
Hugging Face Trending Papers
Jun 3
From Symbolic to Geometric: Enabling Spatial Reasoning in Large Language Models
Recent large language models (LLMs) often appear to exhibit spatial reasoning ability; however, this capability is largely \emph{symbolic}, arising from pattern matching over spatial language rather than true \emph{geometric} reasoning over space. Because LLMs operate on discrete tokens, they lack native support for continuous spatial representations, explicit geometric computation, and structured spatial operators.
arXiv AI
Jun 4
From Symbolic to Geometric: Enabling Spatial Reasoning in Large Language Models
arXiv:2606. 04381v1 Announce Type: cross Abstract: Recent large language models (LLMs) often appear to exhibit spatial reasoning ability; however, this capability is largely \emph{symbolic}, arising from pattern matching over spatial language rather than true \emph{geometric} reasoning over space.
Hugging Face Trending Papers
Jul 9
Understanding Axes of Difficulty For Long Context Tasks Via PredicateLongBench
Large language models (LLMs) have demonstrated rapidly improving long-context capabilities, prompting a wave of benchmarks designed to evaluate them. However, existing long-context evaluations - from Needle-in-a-Haystack (NIAH) tests to more recent multi-hop reasoning and summarization tasks - predominantly measure average-case performance, and many are either saturated or lack robustness.