The paper introduces ShadowBench, a benchmark built from real photographs that tests vision‑language models (VLMs) on three shadow‑related tasks: reading a visible shadow’s direction, inferring where a shadow should fall when absent, and judging physical consistency of shadows. Evaluation of 19 open VLMs and proprietary models shows that while they can read shadow direction well, they perform poorly on inferring absent shadows and judging consistency, with only Gemini‑3.1‑pro slightly above chance. The authors demonstrate that routing a VLM through an image‑generation model (IGM) provides the missing knowledge, significantly improving inference and highlighting the potential of image‑generation objectives for illumination‑aware VLMs.
Commercial vision-language models are reshaping computer vision, with visual priors broad enough to rival task-specific systems. This raises a natural question: do they reduce the need for classic, physics-informed low-level vision?
arXiv:2608. 06075v1 Announce Type: cross Abstract: Commercial vision-language models are reshaping computer vision, with visual priors broad enough to rival task-specific systems.
By Shilin Hu, Jingyi Xu, Dimitris Samaras, Hieu Le
OccludeBench is a manually curated benchmark that tests vision‑language models (VLMs) on their ability to recover fully occluded objects from remaining visual evidence. The benchmark distinguishes direct cases—requiring recovery from a specific physical cue such as a shadow or reflection—from inferential cases, which rely on broader scene context. Across five VLMs, accuracy averages 42.7 % on direct cases versus 62.7 % on inferential cases, a 20‑point gap, while humans achieve 63.4 % and 68.5 % respectively with no significant gap. The study introduces OccludeTrace, a four‑stage diagnostic that isolates the failure point, revealing that models struggle most when recovering the relevant object from the full image projection. Interventions such as inpaint‑then‑confront reduce the gap to a statistically non‑significant difference for models with image‑generation capabilities.
By Jasin Cekinmez, Addison J. Wu, Ryo Mitsuhashi, Linrong Cai, Xingyu Fu, Zhuang Liu
arXiv:2512.06174v3 Announce Type: replace
Abstract: Generating realistic cast shadows for inserted foreground objects requires reasoning about scene geometry and illumination. However, most learning-...
By Shilin Hu, Jingyi Xu, Akshat Dave, Dimitris Samaras, Hieu Le
arXiv:2607. 15565v1 Announce Type: cross Abstract: Where should the question go in a vision-language model (VLM) prompt: before the image or after it?
By Rakshanda Hassan Abhinandan, John Galeotti, Deva Ramanan, Gautam Rajendrakumar Gare
The paper introduces a free, label‑free visual evidence signal that improves fine‑grained vision‑language reasoning. By selecting image crops that maximize the model’s answer distribution peak, the method locates answer‑bearing regions without training or annotations, boosting accuracy from 70 % to 85 %. The evidence gap also complements model confidence, enabling better correctness prediction and error flagging.
By Santi Ram Tiwari, Nihal Naik, Devbrat Pandey, Nishant Sinha
The paper introduces a counterfactual benchmark to evaluate whether vision models understand physical constraints or merely learn rendering shortcuts. Using Mitsuba 3, it generates 5,500 synthetic images across 500 scene families, each with a clean image and ten violations (shadows, reflections, support, surface response, occlusion). Diagnostic baselines (LLaVA‑1.5‑7B, Qwen2.5‑VL‑7B, InternVL3.5‑8B) achieve 64.0–67.8 % macro‑F1 on a held‑out synthetic test set, but performance drops to 40.8 % for LLaVA‑1.5‑7B under an intervention shift, indicating reliance on rendering cues rather than true physical reasoning.
By M. Moein Esfahani, Sepehr Salem, Mohammed Alser, Vince Calhoun
The paper introduces BedBench, a benchmark of thirteen procedurally generated tasks designed to test vision‑language models’ ability to recover and consistently use exact 3D object properties. Across twelve open‑weight and four frontier models, the study finds a stark gap: humans achieve 93.3 % accuracy on per‑color counting, while the best model scores only 7.2 %. Even targeted fine‑tuning can raise a model’s accuracy to 68.6 % but leaves its answers about the most common colors inconsistent with its own per‑color counts, highlighting that recovering a property does not guarantee its consistent use.
By Khai Le-Duc, Nguyen Kim Hai Bui, An Ngo, Nam D. Hoang, Yueqi Song, Jiarui Liu, Thai Quoc Hoang, Fang Wu, Chris Ngo, Terry Jingchen Zhang, Mona T. Diab, Ran Xu, Silvio Savarese, Xiao-Ming Wu
arXiv:2610.29241v1 Announce Type: new
Abstract: In this paper, we investigate why vision-language models (VLMs) produce errors by decomposing them into two sources: visual-representation failures, in...
By Anat Veikherman, Mor Ventura, Nitay Calderon, Lihi Zelnik-Manor, Roi Reichart
The paper introduces visual adaptations of counterfactual tests—vCT and vCCT—to evaluate whether chain-of-thought explanations in vision‑language models faithfully reflect the visual evidence driving predictions. Using these tests, the authors benchmark eight open‑source VLMs on two datasets and find that CoTs often fail to track visual evidence, sometimes omitting removed objects or mentioning them inconsistently. They also release two new datasets, Counter‑SNLI‑VE and Counter‑A‑OKVQA, consisting of image pairs that differ by a single object to facilitate further research.
By Bayar Menzat, Maximilian S\"uss, Ruizhi Wang, Benno Steinegger, Thomas Lukasiewicz, Oana-Maria Camburu
arXiv:2606. 13156v2 Announce Type: replace-cross Abstract: Letting a vision-language model (VLM) think longer at test time has driven much recent progress.
By Animesh Tripathy, Aswanth Krishnan