Multimodal large language models have made rapid progress in video understanding, yet existing benchmarks largely rely on simple prompts and provide limited evidence about whether models can satisfy explicit output constraints. We introduce VCIFBench, a benchmark for evaluating complex instruction following in video understanding.
Video-IFBench is a new benchmark designed to evaluate how well multimodal large language models (MLLMs) follow user-specified instructions in video understanding tasks. It introduces an instruction taxonomy with four templates—single-task, multi-task, selection, and nested—covering 32 task types and 39 constraint categories that span semantic and format requirements. The benchmark was built using a semi-automatic pipeline that combines MLLMs, programmatic processing, and human verification, producing 1.5K samples, and a large-scale evaluation of over 20 recent MLLMs shows that instruction following remains difficult, especially for complex constraints and conditional structures.
arXiv:2503. 06573v3 Announce Type: replace-cross Abstract: Recent LLMs have shown remarkable success in following user instructions, yet handling instructions with multiple constraints remains a significant challenge.
Instruction-following ability is critical for deploying large language models in real-world applications, where downstream components depend on the output satisfying specific constraints. Modern deployments increasingly handle the full task in a single LLM call, with one prompt specifying a layered output whose overall artifact, structural sections, and nested fields must each satisfy concrete constraints.
MM-IFEval-Pro is a new multilingual benchmark for evaluating instruction-following in vision-language models, covering both Chinese and English tasks. It includes 4 major task categories, 24 subcategories, and 8 instruction categories with 52 subcategories, each sample featuring an average of 3.0 constraints to mimic complex instruction scenarios. A reinforcement-learning training set with Chinese and adversarial instructions improves model performance on MM-IFEval-Pro and transfers well to other multimodal benchmarks, showing strong cross-task and cross-language generalization.
arXiv:2606.01600v2 Announce Type: replace-cross Abstract: Video world models are increasingly used in robotic manipulation, yet existing benchmarks mostly evaluate them under valid, feasible, and saf...
VISA (Visual Instruction Synthesis Agent) is an agentic framework that transforms multimodal instruction synthesis into a self‑evolving loop. Each cycle analyzes images to filter constraints, samples new constraint sets, generates candidate instructions, and verifies them using executable tools and large language model judges. Failed samples trigger diagnostic recovery, while accepted samples are evaluated against the target model to estimate difficulty, with all feedback written back to memory to adapt future rounds and provide reward signals for reinforcement learning.
arXiv:2608. 19207v1 Announce Type: new Abstract: Production deployments of Multimodal Large Language Models (MLLMs) increasingly rely on system messages to govern model behavior.
The paper introduces VWG-Bench, a benchmark covering nine reasoning dimensions and 38 tasks to evaluate video generative models on symbolic reasoning, physical laws, and goal pursuit. It also presents Vid-PRE, a prompt-rewriting framework that offloads reasoning to a VLM, improving logical performance without changing the generator architecture. Experiments show that current models excel at visual quality but struggle with logic-heavy tasks, while Vid-PRE significantly boosts reasoning across multiple generators.
Imag‑Eval is a new language‑grounded benchmark for evaluating Text‑to‑Image models, focusing on how well they follow compositional natural‑language instructions. It disentangles prompt length from compositional difficulty by independently varying the number of instances and the combination of constraints (rules), providing 1,140 prompts and 8,842 rule combinations. The study shows that for structured skills, the difficulty is mainly driven by the number of grounded rules and their binding to instances rather than prompt length alone.
RECAST is a new framework that generates datasets with far more constraints per example than existing benchmarks, aiming to push large language models (LLMs) to better follow complex instructions. The authors built RECAST-30K, a 30,000‑instance dataset covering 19 constraint types extracted from real prompt‑response pairs, and showed that fine‑tuning on it improves LLMs’ ability to handle complex tasks without harming general performance. RECAST also provides rule‑based and LLM‑based validators for automatic constraint verification, enabling reward‑based reinforcement learning to further enhance model performance on challenging tasks.
arXiv:2609.10355v1 Announce Type: cross Abstract: Video understanding has rapidly evolved toward video large language models (VideoLLMs): systems that couple video representations with pretrained lar...