AgentVQA is an offline benchmark designed to evaluate vision‑language models on realistic agentic scenarios across five domains: Web Agents, Robotics, Egocentric Videos, Games, and Spatial Planning. It comprises 1,000 carefully selected real‑world examples drawn from 23 source datasets, enabling reliable comparisons without costly online simulations. The benchmark reveals that current models achieve only 58.5% accuracy, with most errors stemming from difficulties in grounding and tracking evolving visual states rather than from task misunderstanding or reasoning.
By Advait Gupta, Sachin Konan, Tom Yin, Gabriel Sarch, Zhuang Liu
UrbanGround is a sandbox that tests how well multimodal large language model agents can translate local street‑view perception into reliable action within a physically realistic replica of Hong Kong. The platform offers closed‑loop first‑person interaction and an interactive map, allowing agents to navigate the 3D city and answer spatial questions. The study evaluates agents across three research questions—scene grounding, navigation over increasing distances, and robustness to route changes—revealing that while agents excel at visual recognition and short‑range reasoning, they struggle with sustained goal‑directed behavior and pedestrian‑aware movement.
By Tianjie Ju, Zheng Wu, Yueqing Sun, Yuhan Cui, Bobo Li, Shengqiong Wu, Pengzhou Cheng, Haodong Zhao, Zongru Wu, Xinbei Ma, Doris Zhang, Kunling Li, Mong-Li Lee, Wynne Hsu, Hao Fei, Qi Gu, Gongshen Liu, Zhuosheng Zhang
arXiv:2608.29880v1 Announce Type: new
Abstract: Open-world geo-localization requires models to reason over ambiguous visual cues through multi-step reasoning and external knowledge grounding. While r...
By Yutian Jiang, Ruijie Li, Sisuo Lyu, Xixuan Hao, Qingxiang Liu, Yongzi Yu, Yuxuan Liang
GTPred is a new benchmark for geo‑temporal prediction that evaluates multi‑modal large language models (MLLMs) on 370 images taken across 120 years worldwide. It assesses predictions by matching both the year and a hierarchical location sequence, and includes annotated reasoning chains to test intermediate reasoning. Experiments on 15 MLLMs show that while visual perception is strong, models still lack world knowledge and geo‑temporal reasoning, and that adding temporal data improves location inference.
By Jinnao Li, Tingzhu Chen, Changbo Wang
arXiv:2610.20055v1 Announce Type: new
Abstract: The rapid advancement of large vision language models (LVLMs) has made geolocation inference from images highly accurate and accessible. This poses ser...
By Yixin Wu, Mingjie Li, Yun Shen, Michael Backes, Yang Zhang
arXiv:2606. 06147v1 Announce Type: new Abstract: End-to-end Vision-Language-Action (VLA) models have shown promise in UAV navigation.
By Shengtao Zheng, Kai Li, Weichen Zhang, Yu Meng, Chen Gao, Xinlei Chen, Yong Li, Xiao-Ping Zhang
SpatialScout is a benchmark that tests vision‑language model agents on active, object‑level 3D mapping. It includes 80 indoor scenes with 3,140 target objects, where agents must control a camera, gather RGB observations, and estimate object centers in a shared coordinate frame. Performance is measured by localization accuracy and target coverage using object‑scale overlap scores, revealing that current VLMs, including Claude Opus 5.5, struggle to build precise metric maps.
By Zhaoxi Zhang, Hao Liang, Linxuan Wu, Yiming Wu, Yifu Yuan, Kun Liang, Chenming Tang, Meiyi Qiang, Wentao Zhang, Weikang Li
MobileWorldBench introduces a benchmark for evaluating vision‑language models as world models in mobile GUI agents, shifting from pixel‑based predictions to natural‑language state transitions. The authors release MobileWorld, a 1.4 million‑sample dataset that enhances VLM world‑modeling performance. They also present a framework that integrates these semantic world models into mobile agent planning, showing improved task success rates.
By Shufan Li, Konstantinos Kallidromitis, Akash Gokul, Yusuke Kato, Kazuki Kozuka, Aditya Grover
arXiv:2603.08491v3 Announce Type: replace
Abstract: Cross-modal Geo-localization (CMGL) matches ground-level text descriptions with geo-tagged aerial imagery, which is crucial for pedestrian navigati...
By Yutong Hu, Jinhui Chen, Chaoqiang Xu, Yuan Kou, Sili Zhou, Shaocheng Yan, Pengcheng Shi, Qingwu Hu, Jiayuan Li
arXiv:2606. 07172v1 Announce Type: cross Abstract: Geospatial understanding is a critical yet underexplored dimension in the development of machine learning systems for tasks such as image geolocation and spatial reasoning.
By Marcelo Sartori Locatelli, Fernando Tonucci, Jea Kwon, Luiz Felipe Vecchietti, Bryan Nathanael Wijaya, Cheng Yaw Low, Virgilio Almeida, Meeyoung Cha
arXiv:2608. 08814v1 Announce Type: cross Abstract: We present 360CityArena, a benchmark for evaluating the urban exploration capabilities of embodied agents within a photorealistic environment constructed from 360-degree videos.
By Kenta Watanabe, Atsuyuki Miyai, Mizuki Takenawa, Kiyoharu Aizawa, Toshihiko Yamasaki
arXiv:2610.18356v1 Announce Type: cross
Abstract: City-scale 3D grounding is commonly formulated as candidate ranking, which requires precomputed instance proposals and becomes inefficient in large o...
By Yu Wang, Rui Dai, Wei Chen, Yi Wang, Bo Dang, Kaikui Liu, Xiangxiang Chu, Yansheng Li