AMIGO (Agentic Multi-Image Grounding Oracle Benchmark) is a long-horizon evaluation framework for vision‑language models that tests hidden‑target identification across galleries of visually similar images. The benchmark requires a model to ask a sequence of attribute‑focused Yes/No questions, receiving Yes/No/Unsure feedback and penalizing invalid actions with Skip, thereby stressing question selection under uncertainty, constraint tracking, and fine‑grained discrimination. Using the Guess My Preferred Dress task, the study shows that final‑answer accuracy alone overstates performance, as models may guess correctly without verified evidence, waste turns, or violate the protocol, highlighting the need for combined visual discrimination, informative questioning, and robust protocol adherence.
By Min Wang, Ata Mahjoubfar
arXiv:2609.39142v1 Announce Type: new
Abstract: Can structured text replace vision for diagram reasoning? A wrong answer after textualization can arise because the representation omits information th...
By Yunbei Zhang, Janet Wang, Jihun Hamm, Chandan K Reddy
arXiv:2608.22174v1 Announce Type: new
Abstract: Unified multimodal models (UMMs) can perform both understanding and generation, raising a central question: can visual generation improve understanding...
By Yubo Zhu, Zhehan Kan, Jingyi Yang, Miaolin Chen, Jinbo Xing, Kai Zhu, Zijian Wang, Sheng Zhong, Wei Tong
Can structured text replace vision for diagram reasoning? A wrong answer after textualization can arise because the representation omits information the question needs, or because the solver fails to...
arXiv:2609.22588v1 Announce Type: new
Abstract: Vision-language models (VLMs) perform strongly on visual question answering benchmarks, yet often make decisions that contradict visual evidence they h...
By Yuyang Dai, Bofei Huang, Hongbo Zhang, Haoran Xie
OccludeBench is a manually curated benchmark that tests vision‑language models (VLMs) on their ability to recover fully occluded objects from remaining visual evidence. The benchmark distinguishes direct cases—requiring recovery from a specific physical cue such as a shadow or reflection—from inferential cases, which rely on broader scene context. Across five VLMs, accuracy averages 42.7 % on direct cases versus 62.7 % on inferential cases, a 20‑point gap, while humans achieve 63.4 % and 68.5 % respectively with no significant gap. The study introduces OccludeTrace, a four‑stage diagnostic that isolates the failure point, revealing that models struggle most when recovering the relevant object from the full image projection. Interventions such as inpaint‑then‑confront reduce the gap to a statistically non‑significant difference for models with image‑generation capabilities.
By Jasin Cekinmez, Addison J. Wu, Ryo Mitsuhashi, Linrong Cai, Xingyu Fu, Zhuang Liu
Large vision-language models can recognize the objects and attributes in a crowded scene yet assign an attribute to the wrong same-class instance. Generic visual-question-answering accuracy marks the...
arXiv:2605.26380v2 Announce Type: replace-cross
Abstract: Frontier multimodal large language models (MLLMs) have been reported to achieve over 90\% accuracy on fine-grained perception benchmarks. How...
By Jingru Chen, Yiming Liu, Mingtao Chen, Sijie Chen, Richeng Xuan, Liang Yang, Zhichao Hu, Fanyang Lu
ReWEIGH the Evidence is a training‑free decoding technique that calibrates token‑level ordinal visual evidence to reduce hallucinations in large vision‑language models. It aggregates vocabulary ranks across visual positions, compares candidates to a token‑specific reference derived from unlabeled images, and applies a bounded penalty only when evidence falls below this reference. Experiments on four 7B backbones show up to a 21.3% reduction in hallucinated object mentions while largely preserving or improving descriptive and general performance, with minimal added latency.
By Jihae Jeong, Junha Choi, Hwanjo Yu
The paper introduces CMPM, a Chinese Multi-Panel Meme benchmark comprising 1,214 annotated samples that capture five structural types, ordering dependencies, panel-order constraints, and optional comment context. It defines a two-layer evaluation: Task 1 tests structure typing and order-sensitive panel sequencing, while Task 2 assesses Chinese meme explanation generation using human ratings across visual, panel, humor, context, and faithfulness dimensions. Benchmarking five large vision‑language models shows that accuracy on canonical displays does not guarantee order understanding, as performance drops sharply under shuffled conditions, and that Gemini 3.1 Pro and GPT‑5.5 outperform open models in Task 2, with comment context providing only modest gains.
By Haihan Li, Haihao Li, Zhenfei Xu, Jize Qian
arXiv:2608. 07435v1 Announce Type: cross Abstract: Vision-language models (VLMs) are improving rapidly, but benchmark development lags behind, making weaknesses hard to identify.
By Zixuan Lan, Luzhe Sun, Matthew R. Walter, Jiawei Zhou
arXiv:2602. 00344v2 Announce Type: replace-cross Abstract: While Retrieval-Augmented Generation (RAG) is one of the dominant paradigms for enhancing Large Vision-Language Models (LVLMs) on knowledge-based VQA tasks, recent work attributes RAG failures to insufficient attention towards the retrieved context, proposing to reduce the attention allocated to image tokens.
By Beidi Zhao, Wenlong Deng, Xinting Liao, Yushu Li, Nazim Shaikh, Yao Nie, Xiaoxiao Li