AVC-Reward introduces an event‑centric framework for evaluating audio‑video consistency, breaking down scenes into individual audio and visual events and assessing them at three levels: global narrative harmony, event‑level temporal synchronization, and fine‑grained attribute alignment. The metric, instantiated as a trainable reward model, is validated against human judgments using the AVC‑Benchmark, which contains diverse positive and negative samples from various generation models. AVC‑Reward outperforms existing automatic methods, offers interpretable failure localization, and enhances consistency when used as a reinforcement‑learning signal.
By Yuruo Wu, Kaihang Pan, Kai Shen, Jiacheng Li, Shuyuan Tu, Jisheng Dang, Li Tuoyu, Juncheng Li, Siliang Tang
arXiv:2608. 18607v2 Announce Type: replace Abstract: Using reinforcement learning to post-train joint video-audio generation models requires a reward signal.
By Yinming Huang, Shuyuan Tu, Xi Yan, Zihan Yang, Jianhua Han, Xu Hang, Yu-Gang Jiang, Zuxuan Wu
AV‑GRPO introduces a modality‑anchored diffusion reinforcement learning framework for joint audio‑video generation, addressing limitations in fidelity, text‑modality alignment, and cross‑modal synchronization. It decouples learning signals through modality‑anchored rollouts, employs trajectory‑locked frozen‑tower optimization to reduce computational cost, and adapts objectives to each modality’s dynamics. The accompanying 5DAV dataset provides difficulty‑controllable, decoupled training samples, and experiments on JavisBench and VABench show AV‑GRPO surpasses LTX‑2.3 in generation quality, semantic alignment, and synchronization.
By Zhiyu Xu, Weilong Yan, Yufei Shi, Shiyang Li, Yihao Liu, Kin-Man Lam, Yuewen Cao
TiTok is an audio‑visual large language model designed for multi‑segment temporal grounding in untrimmed videos. It introduces Time Token Interleaving (TTI) to align temporal perception with prediction, and uses decoupled reinforcement‑learning rewards (global, local, count, precision, format) optimized via GDPO. Evaluated on a new UnAV‑100 protocol with the CountF1 metric, TiTok achieves state‑of‑the‑art results (65.7 mIoU, 0.58 CountF1).
By Eunji Shin, Dahyun Choi, Seungyeon Jo, Yejin Hong, Jiyoung Lee
arXiv:2608.30125v1 Announce Type: cross
Abstract: Current video-to-music (V2M) models lack semantic control and fail to penalize instruction violations, largely due to their reliance on reconstructio...
By Aryan Vijay Bhosale, Vaibhavi Lokegaonkar, Vishnu Raj, Gouthaman KV, Sreyan Ghosh, Ramani Duraiswami, Lie Lu, Dinesh Manocha
arXiv:2605. 07061v2 Announce Type: replace-cross Abstract: Joint audio-video generation models are rapidly approaching professional production quality, raising a central question: do they understand audio-visual physics, or merely generate plausible sounds and frames that violate real-world consistency?
By Zijun Cui, Xiulong Liu, Hao Fang, Mingwei Xu, Jiageng Liu, Zexin Xu, Weiguo Pian, Shijian Deng, Feiyu Du, Chenming Ge, Yapeng Tian