← Back to all news
arXiv AI By Haonan Jiang, Guojian Zhan, Jiancong Xie, Shijun Wan, Dongiia Zhao, Cheng Chen, Yahui Liu, Yao Hu, Chuan Mu

NoteVQA: Benchmarking VLMs on Real-Life Questions from Human Communities

Read the original on arXiv AI →

NoteVQA is a new benchmark that collects 252 real‑life visual questions from the Chinese image‑sharing platform Xiaohongshu, covering 12 topics and 7 user intents. Each question is paired with a concise expert reference and a human‑audited interleaved answer that blends text and visual evidence. The study evaluates VLMs on short‑answer correctness and interleaved answer quality using a new AgenticInterleave framework and a 12‑dimensional IVR‑12 rubric, finding that even state‑of‑the‑art models achieve only about 53% accuracy and lag behind human references in content quality.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Computation and Language
Oct 16

VELD-QA: A Large-Scale Dataset for Visual Evidence Grounding and Question Answering in Long Documents

VELD-QA is a large-scale benchmark for visual evidence grounding and question answering in long documents, comprising 15,000 human-authored question-answer pairs across 5,859 PDFs with 372,445 pages. The dataset includes detailed annotations of supporting pages, visual evidence elements, and descriptions of cited visual content, enabling separate evaluation of retrieval, localization, and interpretation. Experiments on nine vision-language models and various retrieval-augmented systems reveal significant gaps in evidence localization and visual interpretation, with oracle accuracy peaking at 81.7% and enrichment of visual-element annotations narrowing performance gaps by 17–24%.

By Zhivar Sourati, Marianne Menglin Liu, Rongguang Wang, Zheng Wang, Yazhe Hu, Lan Guo, Yuying Wang, Mengqing Guo, Bhavesh Dnyaneshwar Gore, Shubham Pathak, Ramapriya Ranganath, Sandeep Sharma, Tao Sheng, Sujith Ravi, Dan Roth
llmsragagentsnlpmultimodalbenchmarks
More like this →
arXiv Computer Vision
Sep 1

Learning to Search: A Decision-Based Agent for Knowledge-Based Visual Question Answering

arXiv:2604.07146v3 Announce Type: replace Abstract: Knowledge-based visual question answering (KB-VQA) requires vision-language models to understand images and use external knowledge, especially for...

By Zhuohong Chen, Zhenxian Wu, Yunyao Yu, Hangrui Xu, Zirui Liao, Zhifang Liu, Xiangwen Deng, Pen Jiao, Haoqian Wang
llmsragagentsnlpfine-tuningmultimodalbenchmarks
More like this →
arXiv Machine Learning
Jul 7

Incentivizing Vision Language Models to Search for Long Video Question Answering

arXiv:2607. 02959v1 Announce Type: cross Abstract: We introduce VSeek, an agentic framework that transforms long-video question answering (LVQA) from a passive, single-pass perception task into a multi-turn retrieval process.

By Harsh Goel, S P Sharan, Sahil Shah, Minkyu Choi, Joungbin An, Kristen Grauman, Sandeep P. Chinchali
llmsagentsnlpreinforcement-learningbenchmarks
More like this →
arXiv Machine Learning
Sep 7

An Integrated Vision-and-Language Pretraining (VLP) and Visual Question Answering (VQA) model to Automate Nondestructive Evaluation Image Analysis

The paper presents ChatNDE Figure to Caption, an AI system that automates the interpretation of nondestructive evaluation (NDE) images. It combines a Vision‑and‑Language Pretraining (VLP) approach using ResNet50 for visual feature extraction and GPT‑2 for natural‑language captioning, evaluated with BLEU scores. Additionally, a Visual Question Answering (VQA) model is integrated to answer specific queries about the images, enhancing interactivity for field inspectors.

By Mehrdad Shafiei Dizaji, Hoda Azari
llmsnlp
More like this →
arXiv Machine Learning
Jun 15

Self-Evolving Visual Questioner

arXiv:2606. 13929v1 Announce Type: cross Abstract: Vision-language models (VLMs) are typically trained as passive answerers, while their ability to actively ask diverse, non-trivial, visual-centric and grounded questions remains underexplored.

By Yijun Liang, Hengguang Zhou, Ming Li, Lichen Li, Cho-Jui Hsieh, Tianyi Zhou
llmsagentsmultimodal
More like this →