arXiv:2608.31106v1 Announce Type: new
Abstract: Recent video generators often omit audio or synthesize it in a separate stage, limiting reciprocal modeling of visual dynamics and acoustic events. We...
By Jiashu Zhu, Yanhao Zheng, Ruitian Tian, Rujing Dang, Shen Zhang, Bingze Song, Jiachen Lei, Ruimin Lin, Jiahong Wu, Xiangxiang Chu
ViSSFlow is a unified flow‑matching framework that jointly tackles Video‑to‑Sound (V2S) and Visual Text‑to‑Speech (VisualTTS) generation. It incorporates a Diffusion Transformer with a disentangled condition aggregation mechanism, using cross‑attention for semantic cues and self‑attention for dense visual information. The authors also introduce a feature‑level data synthesis strategy to compensate for limited joint training data, and demonstrate that a single model can be trained end‑to‑end without sacrificing performance on either task.
By Xin Cheng, Yuyue Wang, Xihua Wang, Yihan Wu, Kaisi Guan, Yijing Chen, Peng Zhang, Xiaojiang Liu, Meng Cao, Ruihua Song
arXiv:2510. 02916v2 Announce Type: replace-cross Abstract: We propose SALSA-V, a multimodal video-to-audio generation model capable of synthesizing highly synchronized, high-fidelity long-form audio from silent video content.
By Amir Dellali, Luca A. Lanzend\"orfer, Florian Gr\"otschla, Roger Wattenhofer
The paper introduces CAVC, a training‑free, modality‑asymmetric caching framework that speeds up diffusion‑based audio‑visual generation by selectively caching audio denoiser outputs while continuing full video refinement. By detecting when audio denoiser outputs become cache‑tolerant, CAVC bypasses audio denoising and cross‑modal attention in the middle of the sampling trajectory, then restores a few joint steps at the end to correct drift and improve synchronization. Experiments on public benchmarks show 1.10–1.35× wall‑clock acceleration, better visual dynamics and temporal statistics, and competitive audio fidelity and alignment.
By Lifan Jiang, Yibo Zhao, Senlin cheng, Bolei Xu, Xiaobo Li, Can Ye, Dehai Zhao, Deng Cai, Boxi Wu
arXiv:2608.05803v2 Announce Type: replace
Abstract: Recent advances in generative video modeling have enabled diverse generation, reference-based synthesis, extension, and editing, but existing appro...
By Vorch Team, Xiaoyu Chen, Yang Ding, Cong Han, Menglin Han, Yuxin Hong, Jiebo Hou, Zequn Jie, Xiang Li, Jing Liu, Qi Liu, Yulei Lu, Siyuan Luo, Lin Ma, Xin Ma, Yinlong Qian, Peng Shi, Fang Wan, Siqi Wang, Yaohui Wang, Yaole Wang, Yidi Wu, Siqian Yang, Mingyu Yin, Haoran Yu, Gang Yue, Lisai Zhang, Yuting Zhang
Recent advances in generative video modeling have enabled diverse generation, reference-based synthesis, extension, and editing, but existing approaches often rely on fragmented task-specific models. A general model must distinguish heterogeneous target, source, and reference signals to determine what to generate, preserve, or use as guidance, while reducing interference among tasks.