arXiv AI

Can MLLMs Reason over Surround-View Driving Videos? A Multi-View Autonomous Driving QA Benchmark

The paper introduces MVDriveBench, a benchmark for evaluating Multimodal Large Language Models (MLLMs) on surround‑view driving video understanding. It features synchronized multi‑view inputs, spatio‑temporal reasoning tasks, and 12 QA categories spanning perception to decision‑making. Experiments on 23 MLLMs show limited surround‑view reasoning, and closed‑loop tests link better reasoning to improved driving performance.

arXiv AI
Jun 2

From Segments to Scenes: Temporal Understanding in Autonomous Driving via Vision-Language Model

arXiv:2512. 05277v3 Announce Type: replace-cross Abstract: Vision-Language Models (VLMs) are increasingly deployed as the perception and reasoning backbone of autonomous agents acting in the wild, with autonomous driving (AD) being one of the most safety-critical instances.

By Kevin Cannons, Saeed Ranjbar Alvar, Mohammad Asiful Hossain, Ahmad Rezaei, Mohsen Gholami, Alireza Heidarikhazaei, Zhou Weimin, Yong Zhang, Mohammad Akbari
arXiv AI
Jun 4

From Segments to Scenes: Temporal Understanding for Agentic Autonomous Driving via Vision-Language Models

arXiv:2512. 05277v4 Announce Type: replace-cross Abstract: Vision-Language Models (VLMs) are increasingly deployed as the perception and reasoning backbone of autonomous agents acting in the wild, with autonomous driving (AD) being one of the most safety-critical instances.

By Kevin Cannons, Saeed Ranjbar Alvar, Mohammad Asiful Hossain, Ahmad Rezaei, Mohsen Gholami, Alireza Heidarikhazaei, Zhou Weimin, Yong Zhang, Mohammad Akbari
arXiv Computer Vision
Nov 5

DSRS: Teaching MLLMs to Perceive, Reason, and Segment in Complex Driving Scenes

The paper introduces DSRS, a new task that requires multimodal large language models to understand driving scenes, interpret implicit instructions, infer spatial relations, detect risks, and apply traffic rules to produce accurate segmentation masks. To support this, the authors create a benchmark dataset and propose DSRS‑Net, an architecture that integrates geometric features with multimodal fusion to improve reasoning segmentation for automated driving. Experiments show DSRS‑Net outperforms existing methods on driving scene reasoning segmentation and other language‑driven segmentation tasks, and the authors plan to release the code and dataset.

By Sicen Guo, Zhengfei Song, Kuanhsun Chen, Xu Huang, Ping Zhong, Wei Ye, Shaoyi Du, Rui Fan
arXiv Computer Vision
Nov 6

nuReasoning: A Reasoning-Centric Dataset and Benchmark for Long-Tail Autonomous Driving

nuReasoning is a large‑scale, real‑world dataset and benchmark designed to advance reasoning‑centric autonomous driving in long‑tail scenarios. It contains 20,000 twenty‑second clips from multiple cities, each annotated with synchronized multi‑camera images, LiDAR, HD maps, object labels, and human‑verified reasoning labels covering spatial, decision, and counterfactual reasoning. The benchmark supports both reasoning evaluation and planning evaluation, demonstrating that fine‑tuning vision‑language models on nuReasoning improves question answering and that reasoning supervision boosts planning performance even without textual outputs at inference time.

By Zhiyu Huang, Johnson Liu, Rui Song, Xuewei Qi, Yiluan Guo, Zewei Zhou, Hanyin Zhang, Mingxuan Gao, Valeria Xu, Jiali Chen, Yun Zhang, Tianhui Cai, Ruining Yang, Yishan Shen, Jiaqi Ma
arXiv AI
Jul 10

AUTOPILOT VQA: Benchmarking Vision-Language Models for Incident-Centric Dashcam Understanding

arXiv:2607. 08745v1 Announce Type: new Abstract: Recent advances in Vision-Language Models, Large Language Models, and Multimodal Large Language Models have improved autonomous driving tasks such as scene understanding, decision making, trajectory prediction, and visual question answering.

By Siddharth Damodharan, Radhika Gupta, Ali Alshami, Ryan Rabinowitz, Jugal Kalita
arXiv AI
Oct 19

STRIDE: Evaluating Spatiotemporal Reasoning in Driving Edge Cases

STRIDE (SpatioTemporal Reasoning In Driving Edge Cases) is a new benchmark designed to evaluate vision-language models on complex, safety-critical driving scenarios. It includes six task families—spatial perception, spatial understanding, temporal memory, temporal extrapolation, trajectory prediction, and scene-context awareness—using multiple-choice questions and free-form visual question answering that require detailed scene explanations and driving decision justifications. The benchmark focuses on unusual object behaviors and complex traffic interactions, and results show that even the best model, GPT‑6‑Astra, scores only 43.4% on average, highlighting significant gaps in spatiotemporal reasoning.

By Lieqi Liu, Rui gao, Jia-Chen Gu, Wenbo Hu, Zhaobin Mo, Ahmadreza Moradipari, Nejib Ammar, Wei Wang, Nanyun Peng
Hugging Face Trending Papers
Jun 23

UniDrive: A Unified Vision-Language and Grounding Framework for Interpretable Risk Understanding in Autonomous Driving

Recent multimodal large language models (MLLMs) have shown strong potential for autonomous driving scene understanding, yet existing methods still face a fundamental trade-off between temporal reasoning and spatial precision. Models that rely on single-frame or low-resolution inputs often miss small, distant, or partially occluded hazards, while language-centric driving models frequently provide limited grounded evidence for their explanations.

arXiv AI
Jun 16

OmniTraffic: A Controllable Generation Pipeline and Benchmark for Spatio-Temporal Traffic Reasoning

arXiv:2606. 15749v1 Announce Type: cross Abstract: Traffic scene understanding requires models to reason beyond object recognition, including lane topology, multi-view geometry, temporal evolution, and signal-phase semantics.

By Maonan Wang, Zhengyan Huang, Kemou Jiang, Yuhang Fu, Jiayue Zhu, Yuxin Cai, Xingchen Zou, Qiaosheng Zhang, Yi Yu, Ding Wang, Xi Chen, Ben M. Chen, Yuxuan Liang, Zhiyong Cui, Man On Pun, Yirong Chen