arXiv Computer Vision
Oct 23

AgentVQA: A Unified Benchmark for Agentic Visual Understanding

AgentVQA is an offline benchmark designed to evaluate vision‑language models on realistic agentic scenarios across five domains: Web Agents, Robotics, Egocentric Videos, Games, and Spatial Planning. It comprises 1,000 carefully selected real‑world examples drawn from 23 source datasets, enabling reliable comparisons without costly online simulations. The benchmark reveals that current models achieve only 58.5% accuracy, with most errors stemming from difficulties in grounding and tracking evolving visual states rather than from task misunderstanding or reasoning.

By Advait Gupta, Sachin Konan, Tom Yin, Gabriel Sarch, Zhuang Liu
arXiv Computer Vision
Aug 28

UrbanGround: From Local Perception to Spatial Agency in a Real-Scale City

UrbanGround is a sandbox that tests how well multimodal large language model agents can translate local street‑view perception into reliable action within a physically realistic replica of Hong Kong. The platform offers closed‑loop first‑person interaction and an interactive map, allowing agents to navigate the 3D city and answer spatial questions. The study evaluates agents across three research questions—scene grounding, navigation over increasing distances, and robustness to route changes—revealing that while agents excel at visual recognition and short‑range reasoning, they struggle with sustained goal‑directed behavior and pedestrian‑aware movement.

By Tianjie Ju, Zheng Wu, Yueqing Sun, Yuhan Cui, Bobo Li, Shengqiong Wu, Pengzhou Cheng, Haodong Zhao, Zongru Wu, Xinbei Ma, Doris Zhang, Kunling Li, Mong-Li Lee, Wynne Hsu, Hao Fei, Qi Gu, Gongshen Liu, Zhuosheng Zhang
arXiv Computer Vision
Aug 27

GTPred: Benchmarking MLLMs for Interpretable Geo-localization and Time-of-capture Prediction

GTPred is a new benchmark for geo‑temporal prediction that evaluates multi‑modal large language models (MLLMs) on 370 images taken across 120 years worldwide. It assesses predictions by matching both the year and a hierarchical location sequence, and includes annotated reasoning chains to test intermediate reasoning. Experiments on 15 MLLMs show that while visual perception is strong, models still lack world knowledge and geo‑temporal reasoning, and that adding temporal data improves location inference.

By Jinnao Li, Tingzhu Chen, Changbo Wang