VKnowU is a benchmark that tests multimodal large language models (MLLMs) on their grasp of visual knowledge—intuitive, human-like understanding of physical and social principles in videos. The benchmark contains 1,680 questions across 1,249 videos, covering eight core types of visual knowledge, and shows that current state‑of‑the‑art MLLMs still lag behind human performance, especially on world‑centric tasks. To address this gap, the authors release VKnowQA and VideoKnow+, a baseline model that incorporates visual knowledge via a See‑Think‑Answer framework and reinforcement learning, improving performance on VKnowU and related datasets.
By Tianxiang Jiang, Sheng Xia, Yicheng Xu, Linquan Wu, Xiangyu Zeng, Limin Wang, Yu Qiao, Yi Wang
The paper introduces Humanity's Sixth Sense (HSS), a benchmark designed to evaluate intuitive visual reasoning in multimodal large language models. HSS covers a wide range of image and video inputs, organizes them under a structured taxonomy, and pairs each with human‑written prompts that probe implicit temporal, spatial, social, and abstract cues that humans can infer at a glance. Current state‑of‑the‑art models, including GPT‑6‑astra, achieve far lower accuracy than humans (53.6 % vs. 93.1 %), highlighting a significant gap in intuitive visual understanding.
By Xingang Guo, Jing Gu, Brian Jang, Renxiong Wang, Utkarsh Tyagi, Daniel Quigley, Steven Li, David Yan, Daniel Yue Zhang, Darvin Yi, Forrest Huang, HiJae Kim, Tianyi Zhang, Jared Lichtarge, Jihua Huang, Le Xue, Manan Tomar, Qiuyi Richard Zhang, Ruofei Yu, Seth Neel, Yaning Hu, Marcella Valentine, Xinzhe Jiang, Daniel Evans, Chenguang Wang, Dustin Tran, Tong Zhao, Yinfei Yang, Yunzhong He
arXiv:2609.06245v1 Announce Type: cross
Abstract: Multimodal Large Language Models (MLLMs) perform strongly on general visual understanding tasks such as visual question answering, yet they often str...
By Yixin Wan, Tianle Zheng, Kai-Wei Chang
arXiv:2610.19443v1 Announce Type: new
Abstract: Native multimodal pre-training trains a single model from scratch on text and multimodal data, enabling it to understand both language and visual input...
By Haoyuan Wu, Aoqi Wu, Jiani Hu, Bei Yu
GraphVerse is a new benchmark for multimodal large language models that tests visual graph reasoning by combining perception, graph-based visual inputs, and multi-step reasoning. It introduces Graph-centric Image Editing strategies to actively challenge models while preserving semantics, and offers a VGR-Score metric that evaluates reasoning quality beyond simple answer accuracy. Experiments show current models have limitations in VGR and demonstrate the effectiveness of GraphVerse for broader multimodal reasoning tasks.
By Yuanfu Sun, Yuanhang Ren, Kang Li, Chuanhao Ji, Jiaxi Li, Jiajin Liu, Ninghao Liu, Qiaoyu Tan
The paper introduces Percept-V, a dataset of 6,000 program-generated images across 30 domains designed to test simple visual perception skills from the TVPS-4 framework. Experiments show that state‑of‑the‑art multimodal large language models perform poorly compared to humans, especially as image complexity increases, and that fine‑tuning yields only limited generalization to related datasets. The study highlights specific perception skills that remain challenging for current models.
By Samrajnee Ghosh, Ashish Goswami, Naman Agarwal, Hemanshu Garg, Chinmay Mittal, Mausam, Parag Singla
arXiv:2608.21853v1 Announce Type: new
Abstract: Large language models are increasingly moving beyond text processing, adding support for other modalities such as images and audio. While text understa...
By S{\l}awomir Dadas, Micha{\l} Pere{\l}kiewicz, Rafa{\l} Po\'swiata, Ma{\l}gorzata Gr\k{e}bowiec, Bart{\l}omiej Jaworski, Izabela Wo\'zniakowska
The paper introduces TIC‑Bench, a new benchmark for evaluating multimodal large language models on deeply interleaved text‑image contexts. It covers logical, temporal, and spatial association tasks, totaling 2,280 questions across eight specific types. The authors benchmarked ten state‑of‑the‑art MLLMs, finding a significant performance gap versus human experts and highlighting persistent challenges in integrating evidence across interleaved visual and textual inputs.
By Zihao Wang, Xi Xiang, Yuwen Sun, Yingyu Li, Yabo Zhang, Yihan Zeng, Fan Li, Wangmeng Zuo
The paper introduces Auto-Comp, a fully automated, concept-driven pipeline that generates photorealistic compositional benchmarks for vision‑language models. Auto‑Comp creates paired Minimal and Contextual samples for each concept, enabling isolation of core binding abilities from visio‑linguistic complexity. Evaluations across 25 models reveal consistent failures in attribute and relational binding, with context helping relational tasks but hindering attribute tasks due to visual clutter.
By Cristian Sbrolli, Toshihiko Yamasaki, Matteo Matteucci
arXiv:2610.20083v1 Announce Type: new
Abstract: Visual world knowledge is a core capability of multimodal large language models (MLLMs), underpinning knowledge-intensive visual question answering and...
By Chen Chen, Zheming Liang, Jiaqi Wang
Melange is a synthetic visual bias benchmark designed to evaluate social biases in multi‑modal large language models (MLLMs). It extends the language‑only BBQ dataset by generating controlled visual scenarios that clearly depict target attributes, and validates images through automated VQA screening and human inspection. Using Melange, 23 open‑source MLLMs were tested in text‑only, image‑only, and image‑with‑context settings, revealing that switching from text to vision reduces fairness by an average of 0.291 and that models often choose stereotypical responses when biased answers are made.
By Razieh Rezaei, Ehsaneddin Asgary, Masoud Jalili Sabet
The paper introduces EvidenceConnect, a data‑synthesis framework that generates training examples by zooming into related image regions to create questions requiring comparison, aggregation, or chaining of visual evidence. By retaining only reference answers agreed upon by independent vision‑language models and discarding questions that survive region ablation, the authors train base VLMs with RLVR to produce Zooming without Zooming v2 (ZwZ2). Although the synthesized data are not benchmark‑specific, ZwZ2 shows improvements across fine‑grained and spatial perception, embodied spatial reasoning, multimodal long‑context understanding, multimodal agent tasks, and general visual reasoning, demonstrating that strengthening the shared capability of connecting evidence yields more generalizable multimodal perception.
By Lai Wei, Yatai Li, Liangbo He, Jun Lan, Renjie Gu, Linhan Cao, Junkai Lin, Huijia Zhu, Weiqiang Wang, Linghe Kong, Yue Wang, Zhuosheng Zhang, Weiran Huang