arXiv Computer Vision
VibeVideoBench:Evaluating Multimodal Agents on Code-Driven Video Generation
Read the original on arXiv Computer Vision →The Flow has not summarised this story yet — read it at arXiv Computer Vision.
Related stories
arXiv Computer Vision
Oct 28
VibeVideoBench: Evaluating Multimodal Agents on Code-Driven Video Generation
VibeVideoBench is a benchmark that evaluates multimodal agents capable of generating videos by writing code to render each frame and sound. The benchmark contains 239 self‑contained tasks in English and Chinese across 12 categories, requiring agents to produce a finished MP4 with sound within one hour without using any pre‑trained image or video generation model. Evaluation is hierarchical, using program checks, 13k task‑specific rubric checks, and five quality dimensions scored by three VLM judge panels, and it shows that GPT‑6 Astra and Claude Opus 5.5 perform best among nine frontier agents.
Hugging Face Trending Papers
Oct 26
VibeVideoBench: Evaluating Multimodal Agents on Code-Driven Video Generation
Diffusion video models produce striking clips, but they cannot reliably render an exact formula, a correct chart, a rule-consistent game or narration synchronized to on-screen events. A different rout...
Hugging Face Trending Papers
Sep 28
Timeline-Bench: Evaluating Agents on Realistic Video-Editing Tasks, from Raw Footage to Final Cut
Timeline-Bench is a benchmark comprising 56 real video‑editing tasks that require AI agents to transform raw production material into finished videos. Each task includes a brief, source assets, a container, and a set of tests that assess format, content, brief compliance, and quality based on 2,582 blind judgments by 43 video editors. In evaluations, the best agent resolved only 15 of the 56 tasks, and most failures were due to quality tests rather than technical errors.
Hugging Face Trending Papers
Jul 27
FilmBench: A Film-Grade Benchmark for Cinematic Video Generation
Progress in video generation keeps narrowing the visual gap between AI-generated and professionally produced footage, yet most benchmarks still draw prompts from web sources or LLM templates and score them with untrained, generic multimodal models. More fundamentally, their evaluation taxonomies remain rudimentary (overall visual quality, coarse text alignment and temporal smoothness) rather than the professional Cinematic Language criteria by which films are actually made and judged, so they assess basic video plausibility rather than film-grade craft.
arXiv Computer Vision
Oct 2
VideoWeaver: Evaluating and Evolving Skills for Agentic Long Video Generation
VideoWeaver is an agent harness and benchmark for long‑video generation that evaluates and evolves reusable skills. It allows an agent to dynamically compose foundation skills into a workflow based on a single high‑level instruction, rather than following a static pipeline. The benchmark includes 16 task categories and 285 cases with multimodal references, and an evidence‑grounded agent‑as‑judge inspects execution traces and final videos to guide skill evolution, improving both process and output quality.
arXiv AI
Jul 28
FilmBench: A Film-Grade Benchmark for Cinematic Video Generation
arXiv:2607. 24241v1 Announce Type: cross Abstract: Progress in video generation keeps narrowing the visual gap between AI-generated and professionally produced footage, yet most benchmarks still draw prompts from web sources or LLM templates and score them with untrained, generic multimodal models.