arXiv AI
Code2Video-Compass: A Comprehensive Benchmark for General-Purpose Code-Based Video Generation across T2V, I2V, and V2V
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Computer Vision
Oct 2
VTR-Bench: A Systematic Benchmark for Evaluating Visual Text Rendering in Video Generation
VTR-Bench is a new benchmark designed to evaluate how well video generation models render text within scenes. It includes 300 prompts across five real-world scenarios such as advertisements and scientific videos, and uses an automated pipeline with human alignment to assess text fidelity and scene/motion requirements. Experiments on 11 state‑of‑the‑art models show that even the best performer has a word error rate of 0.250, underscoring widespread challenges in visual text rendering.
arXiv Computer Vision
Sep 3
OmniEdit-Bench: A Comprehensive Benchmark for Instruction-based Video Editing
OmniEdit-Bench introduces a comprehensive benchmark for instruction-based video editing (IVE), addressing limitations of existing datasets by covering spatial, temporal, audio, and reference-based editing tasks and distinguishing explicit from implicit instructions. The evaluation framework assesses editing quality across accuracy, preservation, realism, and consistency, using human judgments and vision-language models, and incorporates an accuracy-aware penalty to ensure instruction fidelity. Experiments reveal that current IVE models perform poorly, highlighting the need for improved methods.
arXiv Machine Learning
Oct 12
Vision2Code: A Multi-Domain Benchmark for Evaluating Image-to-Code Generation
Vision2Code is a new benchmark for image-to-code generation that covers 15 domains and 539 test examples without relying on reference source code. It evaluates generated programs by executing them and comparing rendered outputs to the original images using domain‑specific rubrics, yielding scores that correlate better with human judgments than generic metrics. The study shows domain variability in model performance and demonstrates that the scoring method can improve fine‑tuning of models like Qwen3.5‑9B and Phi‑4‑MM via rejection sampling.
arXiv Computer Vision
Aug 27
VGA-BenchV2: An Expanded Unified Benchmark and Multi-Model Framework for Evaluating Video Aesthetics and Generation Quality
VGA‑BenchV2 is an expanded, human‑aligned benchmark and optimization framework that jointly evaluates video generation quality and aesthetic value. It builds on the original VGA‑Bench taxonomy, adding 52 sub‑dimensions and 1,016 curated prompts to generate over 60,000 videos from 12 mainstream models. The benchmark significantly enlarges human supervision with 36,000 task‑level annotations and introduces a hybrid evaluator (VAQA‑Net, VTag‑Net, VGQA‑Net) that aligns well with human judgments and can be used as a reward model for reinforcement‑learning fine‑tuning.
arXiv Computer Vision
Sep 21
OmniVBench: A Benchmark and Large-Scale Dataset for Omni Reference-to-Video Generation
OmniVBench introduces a comprehensive benchmark and a large-scale dataset for omni reference-to-video (R2V) generation, addressing gaps in existing evaluations that focus only on limited reference types and holistic consistency. The benchmark expands evaluation across 7 task families and 18 fine-grained tasks, covering content, motion, style, structure, narrative, and multi-reference settings, and employs a factor‑grounded evaluation with 12,172 checklist items to assess preservation, disentanglement, and routing of reference factors. The accompanying Omni‑R2V Dataset provides 340K training samples derived from professional video footage, along with task‑specific pipelines for scalable data construction, enabling broader research and revealing performance gaps in current R2V models.
Hugging Face Trending Papers
Aug 26
VGA-BenchV2: An Expanded Unified Benchmark and Multi-Model Framework for Evaluating Video Aesthetics and Generation Quality
We introduce VGA-BenchV2, an extended human-aligned benchmark and optimization framework for jointly evaluating and improving video generation quality and aesthetic value. Built upon VGA-Bench, VGA-Be...