The paper introduces Cue2Narrate, a two‑stage pipeline that jointly predicts what visual events to narrate and when to insert the narration in long, untrimmed movie clips. It uses a dual‑head audio‑visual localizer to identify visual cue and narration windows, followed by a LoRA‑adapted vision‑language model that generates concise audio descriptions, trained with a Description Ranking Loss. The authors also present the LongLSMDC benchmark, comprising up to 8‑minute clips, and show that Cue2Narrate outperforms video‑only and audio‑only baselines by 5–12 points in average mAP and improves AD generation over fine‑tuned base VLMs.
By Akshita Gupta, Aditya Arora, Federico Tombari, Marcus Rohrbach, Anna Rohrbach
Audio Description (AD) provides spoken narration of visual events during dialogue gaps, making movies accessible to visually impaired audiences. The problem requires determining both what (which visua...
The paper introduces Temporal Context Routing (TCR), a method that aligns script timing with the shared temporal axis used for audio and video generation. By routing prompt guidance to specific temporal positions in both modalities, TCR dramatically improves shot boundary accuracy and dialogue timing on a set of test scripts, while preserving visual quality and audio‑visual sync. A user study confirms that participants prefer TCR across multiple evaluation dimensions.
By Yichen Liu, Quanwei Zhang, Haozhe Wang, Donghao Zhou, Xiaojie Li, Yang Shi, Jiaming Liu, Ruihua Huang, Yingtian Zou, Daquan Zhou
arXiv:2609.10355v1 Announce Type: cross
Abstract: Video understanding has rapidly evolved toward video large language models (VideoLLMs): systems that couple video representations with pretrained lar...
By Killian Steunou, Yannis Tevissen, Moun\^im A. El Yacoubi
WanPE is a 397‑B parameter prompt‑enhancement model that learns director‑level cinematic planning from 1.05 M real‑world videos. It generates shot‑level cinematic plans through video‑grounded reverse construction and uses Semantic‑Consistency GRPO (SC‑GRPO) to maintain user intent across shots and time. In evaluations, WanPE improves human preference over raw prompts by up to 50.86 points for 30‑second videos and outperforms commercial offerings for shorter durations.
By Yubo Zhu, Yawen Shao, Ziyun Dai, Zixun Fang, Kai Zhu, Siyang Sun, Haolan Xue, Chuxin Wang, Tingyu Weng, Jingming Luo, Chen Shi, Lianghua Huang, Yufeng Ai, Yuzheng Wang, Wenyuan Zhang, Yu Shang, Yuxiang Bao, Zoubin Bi, Jie Xiao, Jinbo Xing, Jiaxing Zhao, Chongyang Zhong, Hengjian Chen, Chenwei Xie, Akide Liu, Zhehan Kan, Yu Liu, Wei Zhai, Sheng Zhong, Wei Tong
Video understanding has rapidly evolved toward video large language models (VideoLLMs): systems that couple video representations with pretrained large language models and condition generation on a te...
arXiv:2608.23383v1 Announce Type: new
Abstract: Video generation is progressing beyond isolated clips toward long-form narratives and interactive worlds, requiring models to preserve identities, foll...
By Nan Duan, Haoyang Huang, Weiyang Jin, Haoran Li, Yaowei Li, Yuming Li, Yijun Liu, Xin Lu, Xiaoxiao Ma, Yanwen Ma, Yaofeng Su, Yilang Sun, Haoyu Wang, Zeyue Xue, Songchun Zhang, Junhao Zhuang
arXiv:2609.13830v1 Announce Type: new
Abstract: We present DiVA, a deeply interactive digital life simulator pioneering a new paradigm for long-term, open-ended interactive experiences within digital...
By Cheng Chen, Hao Ouyang, Qiuyu Wang, Ka Leong Cheng, Wen Wang, Yihao Meng, Hanlin Wang, Yixuan Li, Jiacheng Wei, Zhenshan Tan, Yanhong Zeng, Yujun Shen, Guosheng Lin, Fayao Liu
The paper introduces an audio-first triage method for budgeted vision‑language captioning of untrimmed egocentric video. By selecting windows for a vision‑language model based on lightweight audio features before any video decoding, the approach reduces costly model calls. It achieves 4.0–10.8 percentage point improvements in action coverage across call rates, cuts 9–20% of VLM calls at matched coverage on EPIC‑KITCHENS‑100, and outperforms uniform sampling and recent visual keyframe selectors on Ego4D.
By Masoud Jalayer, Changyi Li, Yu Xiao
Video generation is progressing beyond isolated clips toward long-form narratives and interactive worlds, requiring models to preserve identities, follow user controls, and remain stable over extended...
The paper introduces Temporally-Grounded Language Generation (TGLG), a benchmark that tests vision‑language models on their ability to produce semantically accurate and temporally precise utterances in real‑time settings. It identifies perceptual updating and contingency awareness as key capabilities, curates datasets from sports broadcasting and egocentric interactions, and proposes the TRACE metric to jointly evaluate semantic similarity and temporal alignment. The authors also present VLM‑TSI, a model that interleaves visual and linguistic tokens in a time‑synchronized manner, achieving better performance than a strong baseline yet still showing modest overall results, underscoring the challenge of real‑time VLMs.
By Keunwoo Peter Yu, Joyce Chai
arXiv:2609.31810v2 Announce Type: replace-cross
Abstract: Video-to-music models have advanced considerably in the last few years, particularly in film and music video applications. In this paper, we...
By Felipe Marra, Lucas N. Ferreira