arXiv AI

SemComp-Bench: Benchmarking Semantic Task Completion in Video Generation

SemComp-Bench introduces a new video generation task called Semantic Task Completion, where a model must produce a video that achieves a specified outcome while maintaining semantic alignment with a reference image. The benchmark includes the SemComp-Data dataset, spanning six domains, and a four-stage curation pipeline that transforms raw videos into standardized instances. Evaluation is performed via a vision‑language model that answers structured binary questions, yielding Outcome Achievement (OA) and Generation Reliability (GR) scores.

arXiv Computer Vision

Aug 27

Video-IFBench: Evaluating Instruction Following of Multimodal LLMs in Video Understanding Scenarios

Video-IFBench is a new benchmark designed to evaluate how well multimodal large language models (MLLMs) follow user-specified instructions in video understanding tasks. It introduces an instruction taxonomy with four templates—single-task, multi-task, selection, and nested—covering 32 task types and 39 constraint categories that span semantic and format requirements. The benchmark was built using a semi-automatic pipeline that combines MLLMs, programmatic processing, and human verification, producing 1.5K samples, and a large-scale evaluation of over 20 recent MLLMs shows that instruction following remains difficult, especially for complex constraints and conditional structures.

By Hongbo Liu, Peixian Chen, Sihan Liu, Peiyuan Zhang, Kai Zou, Dian Zheng, Xiaoxing Hu, Yuhao Dong, Mengdan Zhang, Yunhang Shen, Haoyu Cao, Wei Liu, Weibo Gu, Xing Sun, Shengjie Zhao

llms multimodal benchmarks

arXiv Computer Vision

Sep 21

OmniVBench: A Benchmark and Large-Scale Dataset for Omni Reference-to-Video Generation

OmniVBench introduces a comprehensive benchmark and a large-scale dataset for omni reference-to-video (R2V) generation, addressing gaps in existing evaluations that focus only on limited reference types and holistic consistency. The benchmark expands evaluation across 7 task families and 18 fine-grained tasks, covering content, motion, style, structure, narrative, and multi-reference settings, and employs a factor‑grounded evaluation with 12,172 checklist items to assess preservation, disentanglement, and routing of reference factors. The accompanying Omni‑R2V Dataset provides 340K training samples derived from professional video footage, along with task‑specific pipelines for scalable data construction, enabling broader research and revealing performance gaps in current R2V models.

By Wenxue Li, Peiyan Guan, Haoyang Jiang, Junxian Cai, Hualuo Liu, Chunjie Zhang, Chong Guan, Songlian Li, Taiyi Wu, Yongjian Yu, Xiaotong Zhao, Alan Zhao, Eric Liu, Xi Chen, Yu Liu, Lei Zhu

diffusion benchmarks

arXiv Computer Vision

Aug 27

AdaVDR: Adaptive Tool Use and Reflection for Video Deep Research

AdaVDR is an adaptive video deep research agent that selects and reflects on tool usage based on the task and the model’s capabilities. It constructs a specialized data pipeline to generate high‑quality QA pairs and uses model‑conditioned filtering to remove unnecessary tool calls. The agent is trained with supervised fine‑tuning and reinforcement learning, achieving top performance on the VDR‑EE benchmark and significant gains on VideoDR.

By Xintong Zhang, Xiaomeng Fan, Shilin Yan, Ekko He, Zicheng Liu, Zijian Zou, Guannan Zhang, Yuwei Wu, Zhi Gao, Hongwei Xue

agents reinforcement-learning fine-tuning benchmarks

arXiv Computer Vision

4d ago

Complementary Retrieval-Augmented Prompting for Consistent Long-Form Video Generation

arXiv:2609.37407v1 Announce Type: new Abstract: While recent video foundation models excel at generating high-quality short videos, long-form video generation remains a critical challenge, where a ma...

By Xianghan Wei, Xiaoda Yang, Zhi Wang, An Pan, Daoan Zhang, Huayi Zhang, Yan Zhang, Wei Xu, Zishun Liao, Jianwen Lou

llms rag agents diffusion multimodal

arXiv AI

Jul 14

Evidence-Backed Video Question Answering

arXiv:2607. 11862v1 Announce Type: cross Abstract: Current Video Large Language Models (Video LLMs) excel in question answering (QA) but largely operate as black boxes, providing textual answers without verifiable visual grounding.

By Shijie Wang, Honglu Zhou, Ziyang Wang, Ran Xu, Caiming Xiong, Silvio Savarese, Chen Sun, Juan Carlos Niebles

llms computer-vision nlp fine-tuning benchmarks

arXiv Computer Vision

Aug 25

Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videos

arXiv:2501.04001v4 Announce Type: replace Abstract: This work presents Sa2VA, the first comprehensive, unified model for dense grounded understanding of both images and videos. Unlike existing multi-...

By Haobo Yuan, Xiangtai Li, Tao Zhang, Yueyi Sun, Zilong Huang, Shilin Xu, Shunping Ji, Yunhai Tong, Lu Qi, Jiashi Feng, Ming-Hsuan Yang

llms computer-vision fine-tuning multimodal benchmarks

arXiv Machine Learning

Sep 1

SynMulti: Synthetic-to-Real Learning for Multimodal Video Understanding

arXiv:2604.12335v2 Announce Type: replace-cross Abstract: Training multimodal large language models (MLLMs) for video understanding requires large-scale annotated data spanning diverse tasks such as...

By Tanzila Rahman, Renjie Liao, Leonid Sigal

llms computer-vision nlp fine-tuning multimodal

arXiv AI

Aug 11

Open Evaluation Agent: Efficient and Promptable Evaluation of Visual Generative Models

arXiv:2608. 09666v1 Announce Type: new Abstract: Recent advances in visual generative models have enabled high-quality image and video generation, but evaluating these models often demands sampling hundreds or thousands of images or videos, which is computationally expensive.

By Shulin Tian, Ziqi Huang, Fan Zhang, Hongyuan Zhu, Yu Qiao, Ziwei Liu

agents diffusion benchmarks

arXiv AI

Jun 24

EG-VQA: Benchmarking Verifiable Video Question Answering with Grounded Temporal Evidence

arXiv:2606. 24797v1 Announce Type: cross Abstract: Recent advances in Video Large Language Models (Video-LLMs) have yielded promising performance on video question answering (VideoQA).

By Linpeng Huang, Weixing Chen, Zexin Chen, Yang Liu, Liang Lin

llms nlp benchmarks safety

arXiv Computer Vision

2d ago

Memory-Guided B-Roll Generation from User Video Collections

arXiv:2610.01884v1 Announce Type: new Abstract: We introduce an approach for collection-grounded B-roll sequence generation. Given a user's video collection, a directive given in natural language, an...

By Cusuh Ham, Fabian Caba Heilbron, Josef Sivic, Bryan Russell

diffusion safety

arXiv AI

Aug 11

RAVEN-Eval: Rubric-Guided Automatic Evaluation for AI Video Generation Models Based on LMM Preference Judgement

arXiv:2608. 09111v1 Announce Type: new Abstract: AI video generation has advanced rapidly and entered widespread commercial use.

By Ziheng Jia, Jiaying Qian, Zicheng Zhang, Xiaorong Zhu, Lancheng Gao, Xiongkuo Min

diffusion benchmarks

arXiv AI

Aug 25

VGI-Bench: Probing Visual Intelligence in Video Generation Models

arXiv:2608.19583v2 Announce Type: replace-cross Abstract: Recent studies suggest that video generation models can exhibit certain forms of zero-shot visual reasoning through generated frames. Yet rel...

By Xuan He, Cong Wei, Yuhao Cheng, Linrui Ma, Yuxuan Zhang, Zuojun Li, Yuhao Wen, Jize Jiang, Zeyi Liu, Yuren Hao, Songcheng Cai, Keming Wu, Penghui Du, Kai Zou, Rui Yang, Chenkai Sun, Ke Yang, Ping Nie, Kelsey R Allen, Chenglong Wang, Michel Galley, Jianfeng Gao, ChengXiang Zhai

diffusion fine-tuning benchmarks

Related stories