arXiv AI By Chao Shen, Xinyuan Li, Yunfan Zhou, Jianguo Yao, Haibing Guan, Zhihai Wang, Xijun Li

InSituMeasure: Probing Situated Measurement Grounding in Industrial Scenes with Multimodal Large Language Models

Read the original on arXiv AI →

InSituMeasure is a new benchmark that tests multimodal large language models on situated measurement tasks in industrial scenes. It includes 2,922 real monitoring images of eight types of engineering instruments, with detailed gauge-attribute labels and noise tags for failure diagnosis. The benchmark defines metrics for numerical accuracy, unit consistency, rejection of unanswerable queries, and alignment of model failures with annotated error factors, revealing that even the best models achieve only 25.7% joint value‑unit accuracy and 51.8% confidence‑diagnosis F1.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv Computer Vision
Sep 3

Beyond Appearance: Can Multimodal Large Language Models Exploit Vertical Structure for Remote Sensing Natural Scene Understanding?

The paper introduces VertiCue-Bench, a diagnostic benchmark designed to test whether multimodal large language models (MLLMs) can perceive, ground, and utilize vertical structure information in remote-sensing natural scenes. It presents a three-stage framework—Perception, Grounding, Utilization—and a Representation Intervention Spectrum across various modalities to evaluate ten state-of-the-art models. The study finds a significant Vertical Structure Utilization Gap: while models show some geometric perception, they struggle to accurately link vertical evidence to spatial entities and incorporate it into semantic decisions.

By Jing Huang, Duanchu Wang, Junjie Yang, Zihang Cheng, Cheng Li, Lin Cui, Zhouyi Wu, Di Wang
Hugging Face Trending Papers
Jul 6

SteelBench: Evaluating Vision-Language Models in Real-World Industrial Environments

Existing video benchmarks evaluate action recognition on consumer videos, egocentric recordings, or simulated industrial environments. They do not test vision-language models under the visual and procedural conditions of real industrial CCTV, where workers appear as distant figures amid dust, steam, low light, glare, occlusion, and overlapping activities.