arXiv:2607. 15418v1 Announce Type: new Abstract: We introduce DrawingVQA, the first benchmark designed to evaluate multimodal large language models (MLLMs) on real-world construction drawings -- a core media in architecture, civil, and many other engineering practices.
By Yoonhwa Jung, Junryu Fu, Mani Golparvar-Fard
Sci‑MMR is a new benchmark for multi‑step evidence‑grounded scientific reasoning in multimodal agents, featuring 235 multi‑hop tasks across four disciplines and an average of nine figure panels per task. It evaluates not just final answer accuracy but also the recovery of structured evidence from scientific claims, citations, visual data, and supporting regions. Experiments on eight state‑of‑the‑art models show a gap of over 20 points between answer accuracy and complete evidence recovery, highlighting significant challenges in evidence acquisition and integration.
By Jiaqiang Li, Yajie Yang, Zhiheng Xi, Jiadong Chen, Enyu Zhou, Senjie Jin, Yang Nan, Jiazheng Zhang, Han Wang, Yanxin Li, Dingwei Zhu, Bicheng Deng, Yuhui Wang, Xiang Zheng, Qi Zhang, Lei Bai, Xingjun Ma, Tao Gui
The paper introduces CAIT, a benchmark of 400 synthetic scenes featuring counter‑intuitive actions that challenge multimodal large language models (MLLMs). Human participants and proprietary models like Claude and Gemini perform well, but standard open‑source instruction‑tuned MLLMs fail, largely due to a strong language prior that overrides contradictory visual evidence. The study shows that Chain‑of‑Thought reasoning can help but introduces new issues, while targeted fine‑tuning and structured prompting can reduce reliance on language priors and improve visual grounding.
By Chen Ling, Tongwei Zhang, Hanqian Li, Nai Ding
MechReason is a new benchmark for multi-image, multi-hop reasoning in mechanical engineering, featuring 12,000 question-answer pairs with explicit reasoning chains and 21,000 visual items across nine evidence types. It covers eight task types—explanation, prediction, design, and diagnosis—within four reasoning dimensions, and is constructed through a four-stage pipeline that extracts engineering claims, generates verification-masked questions, and validates multimodal quality. Even state‑of‑the‑art models score only 62.89% on this dataset, highlighting its difficulty.
By Tengyue Wang, Kang An, Chenxu Du, Zhongyu Yang, Yuanchi Zhu, Xinqi Yang, Hebao Zhu, Ziliang Wang, FaQiang Qian, Yunli Yang, Qibing Ren
Mr.LHDR is a new benchmark designed to evaluate deep research agents on long‑horizon, multimodal tasks. It presents questions built from hidden Node‑Relation graphs that require an average of 12.1 intermediate conclusions and a mean dependency depth of 10.4 before arriving at a single verifiable answer. The benchmark tests both final answers and the correctness of intermediate conclusions, using metrics such as Overall Accuracy, Strict Accuracy, Checklist Score, and Dependency‑Aware Checklist Score.
By Minghao Guo, Meng Cao, Sui Zhao, Siyu Ning, Xin Wang, Haoze Zhao, Jiaxuan Yang, Haihong Hao, Mingfei Han, Shunlin Rong, Haijun Wu, Xiaodan Liang, Xiaojun Chang
arXiv:2606. 03273v2 Announce Type: replace-cross Abstract: Visual DeepSearch tasks require multimodal large language models (MLLMs) to resolve complex visual queries by repeatedly inspecting image regions, grounding reasoning in visual evidence, and connecting fine-grained clues across multiple steps.
By Hang He, Chuhuai Yue, Chengqi Dong, Chengcheng Wan, Ting Su, Haiying Sun, Jiajun Chai, Xiaohan Wang, Guojun Yin