arXiv:2608.31106v1 Announce Type: new
Abstract: Recent video generators often omit audio or synthesize it in a separate stage, limiting reciprocal modeling of visual dynamics and acoustic events. We...
By Jiashu Zhu, Yanhao Zheng, Ruitian Tian, Rujing Dang, Shen Zhang, Bingze Song, Jiachen Lei, Ruimin Lin, Jiahong Wu, Xiangxiang Chu
ViSSFlow is a unified flow‑matching framework that jointly tackles Video‑to‑Sound (V2S) and Visual Text‑to‑Speech (VisualTTS) generation. It incorporates a Diffusion Transformer with a disentangled condition aggregation mechanism, using cross‑attention for semantic cues and self‑attention for dense visual information. The authors also introduce a feature‑level data synthesis strategy to compensate for limited joint training data, and demonstrate that a single model can be trained end‑to‑end without sacrificing performance on either task.
By Xin Cheng, Yuyue Wang, Xihua Wang, Yihan Wu, Kaisi Guan, Yijing Chen, Peng Zhang, Xiaojiang Liu, Meng Cao, Ruihua Song
arXiv:2510. 02916v2 Announce Type: replace-cross Abstract: We propose SALSA-V, a multimodal video-to-audio generation model capable of synthesizing highly synchronized, high-fidelity long-form audio from silent video content.
By Amir Dellali, Luca A. Lanzend\"orfer, Florian Gr\"otschla, Roger Wattenhofer
The paper introduces CAVC, a training‑free, modality‑asymmetric caching framework that speeds up diffusion‑based audio‑visual generation by selectively caching audio denoiser outputs while continuing full video refinement. By detecting when audio denoiser outputs become cache‑tolerant, CAVC bypasses audio denoising and cross‑modal attention in the middle of the sampling trajectory, then restores a few joint steps at the end to correct drift and improve synchronization. Experiments on public benchmarks show 1.10–1.35× wall‑clock acceleration, better visual dynamics and temporal statistics, and competitive audio fidelity and alignment.
By Lifan Jiang, Yibo Zhao, Senlin cheng, Bolei Xu, Xiaobo Li, Can Ye, Dehai Zhao, Deng Cai, Boxi Wu
arXiv:2608.05803v2 Announce Type: replace
Abstract: Recent advances in generative video modeling have enabled diverse generation, reference-based synthesis, extension, and editing, but existing appro...
By Vorch Team, Xiaoyu Chen, Yang Ding, Cong Han, Menglin Han, Yuxin Hong, Jiebo Hou, Zequn Jie, Xiang Li, Jing Liu, Qi Liu, Yulei Lu, Siyuan Luo, Lin Ma, Xin Ma, Yinlong Qian, Peng Shi, Fang Wan, Siqi Wang, Yaohui Wang, Yaole Wang, Yidi Wu, Siqian Yang, Mingyu Yin, Haoran Yu, Gang Yue, Lisai Zhang, Yuting Zhang
Recent advances in generative video modeling have enabled diverse generation, reference-based synthesis, extension, and editing, but existing approaches often rely on fragmented task-specific models. A general model must distinguish heterogeneous target, source, and reference signals to determine what to generate, preserve, or use as guidance, while reducing interference among tasks.
arXiv:2608.05776v2 Announce Type: replace
Abstract: Autoregressive continuation provides a natural path toward minute-scale audio-visual generation by repeatedly extending a short-window generator co...
By Lisai Zhang, Yidi Wu, Qi Liu, Xin Ma, Yang Ding, Gang Yue, Siqian Yang, Jingyuan Chen, Lin Ma, Yaohui Wang
AV-FlowEdit is a training‑free, inversion‑free framework that enables prompt‑driven audio‑visual editing using joint audio‑video diffusion models. It leverages source‑to‑target flow differences to guide edits while preserving non‑target structure and cross‑modal correspondence. The method includes a Cross‑Modal Event‑Aware Spectral Trajectory Preservation module that suppresses drift by combining flow saliency, aligned audio activity, multiscale low‑frequency displacement, and diffusion‑stage scheduling, achieving strong prompt alignment, temporal consistency, and competitive audio quality.
By Shaohua Pan, Xiao Liu
arXiv:2610.00691v1 Announce Type: new
Abstract: Recent joint audio-video generative models can synthesize realistic videos with synchronized sound, but typically generate audio as a single mixed trac...
By Zhuo Ning, AmirHossein Naghi Razlighi, Sagi Polaczek, Daniel Cohen-Or, Ali Mahdavi-Amiri
Long audio-video reasoning is difficult for omnimodal LLMs because the decisive evidence is often sparse, cross-modal, and too expensive to preserve with uniformly high-fidelity inputs. We introduce OmniReasoner, a tool-use post-training framework for Thinking with Long Audio-Video: omni-modal LLMs learn, via supervised fine-tuning and reinforcement learning, to decide whether and where to call a zoom-in tool before answering.
arXiv:2608.05663v3 Announce Type: replace
Abstract: Real-time long-form avatar audio-video generation requires causal, continuous synthesis while maintaining audiovisual synchronization and visual co...
By Menglin Han, Yang Ding, Yulei Lu, Haoran Yu, Xin Ma, Junyi Chen, Zhangkai Ni, Lin Ma, Yaohui Wang
arXiv:2602. 12304v5 Announce Type: replace-cross Abstract: Existing mainstream video customization methods focus on generating identity-consistent videos based on given reference images and textual prompts.
By Maomao Li, Zhen Li, Kaipeng Zhang, Guosheng Yin, Zhifeng Li, Dong Xu