← Back to all news
arXiv Machine Learning By Sourajit Saha, Shubhashis Roy Dipta, Nobin Sarwar, Shaswati Saha, Yuxuan Jiang, Siyuan Li, Qiheng Wang

New Evidence, Same Choice: Testing Physical Experiment Selection in Vision Language Models

Read the original on arXiv Machine Learning →

The paper introduces a new benchmark for vision‑language models that tests their ability to decide whether to answer a physics question immediately or to request additional experimental evidence. Each problem presents one measurement image and four possible physical worlds defined by two masses and two values of another property; the model must either stop and answer or choose the cheapest experiment that resolves the question. Across six open models and 144 parameter sets, the models almost always repeat the same action even when the optimal choice changes, and only a single model gets both decisions correct on 5.9% of cases.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Computer Vision
Oct 7

World Models' Last Exam in Physics

arXiv:2610.08791v1 Announce Type: new Abstract: Video world models can produce visually convincing yet physically inconsistent sequences, raising concerns about their reliability for prediction and p...

By Mingju Gao, Qingle Liu, Yuzhao Peng, Xinjie Lin, Ziming Qin, Zheng Jiang, Wenyi Li, Calvin Xiao, Youjie Zheng, Kaisen Yang, Qinhuai Na
llmsdiffusionroboticsmultimodalbenchmarks
More like this →
arXiv AI
Oct 27

InfiniteWorld: Separating Visual Evidence from Familiar Physics

InfiniteWorld is a new benchmark that pairs physical‑reasoning questions with simulator videos whose physical parameters are systematically altered, allowing researchers to test whether models rely on visual evidence or on familiar physics. The benchmark’s generation pipeline connects simulation, scene screening, paired video translation, question construction, and input checks, and it includes five tasks—law identification, parameter estimation, temporal reasoning, analytic counterfactual comparison, and law induction. Evaluation across five matched input settings (text‑only, ordered, shuffled observations) reveals that video‑language models often default to Earth‑normal physics, showing strong accuracy on normal questions but poor recognition of altered dynamics. "whyItMatters":"The benchmark provides a measurable way to separate visual evidence from familiar physics, exposing persistent biases in current video‑language models toward Earth‑normal rules."

By Tong Zhang, Yuxuan Cheng, Yun Peng
llmsbenchmarks
More like this →