arXiv Computer Vision

Encore: Infinite Audio-Video Generation with Adaptive Signal Routing

Encore is a new framework for generating long, synchronized audio‑video content. It splits the problem into local continuity, handled by iterative chunk‑wise synthesis with cross‑chunk context, and global consistency, enforced through reference audio‑video signals with shifted position embeddings. The Adaptive Signal Routing mechanism learns attention biases and residual scales to modulate the influence of each conditioning signal, enabling end‑to‑end joint audio‑video generation and infinite‑length inference.

arXiv Computer Vision
Sep 1

DreamX-Creator: Democratizing Native Audio-Video Generation at 2K Resolution

arXiv:2608.31106v1 Announce Type: new Abstract: Recent video generators often omit audio or synthesize it in a separate stage, limiting reciprocal modeling of visual dynamics and acoustic events. We...

By Jiashu Zhu, Yanhao Zheng, Ruitian Tian, Rujing Dang, Shen Zhang, Bingze Song, Jiachen Lei, Ruimin Lin, Jiahong Wu, Xiangxiang Chu
arXiv Computation and Language
Nov 4

ViSSFlow: Unifying Video-Conditioned Sound and Speech Generation via Joint Learning

ViSSFlow is a unified flow‑matching framework that jointly tackles Video‑to‑Sound (V2S) and Visual Text‑to‑Speech (VisualTTS) generation. It incorporates a Diffusion Transformer with a disentangled condition aggregation mechanism, using cross‑attention for semantic cues and self‑attention for dense visual information. The authors also introduce a feature‑level data synthesis strategy to compensate for limited joint training data, and demonstrate that a single model can be trained end‑to‑end without sacrificing performance on either task.

By Xin Cheng, Yuyue Wang, Xihua Wang, Yihan Wu, Kaisi Guan, Yijing Chen, Peng Zhang, Xiaojiang Liu, Meng Cao, Ruihua Song
arXiv Computer Vision
Oct 23

CAVC: Convergence-Adaptive Audio-Visual Cache for Modality-Asymmetric Diffusion Generation

The paper introduces CAVC, a training‑free, modality‑asymmetric caching framework that speeds up diffusion‑based audio‑visual generation by selectively caching audio denoiser outputs while continuing full video refinement. By detecting when audio denoiser outputs become cache‑tolerant, CAVC bypasses audio denoising and cross‑modal attention in the middle of the sampling trajectory, then restores a few joint steps at the end to correct drift and improve synchronization. Experiments on public benchmarks show 1.10–1.35× wall‑clock acceleration, better visual dynamics and temporal statistics, and competitive audio fidelity and alignment.

By Lifan Jiang, Yibo Zhao, Senlin cheng, Bolei Xu, Xiaobo Li, Can Ye, Dehai Zhao, Deng Cai, Boxi Wu
arXiv Computer Vision
Oct 27

Vorch-Omni: Multi-Task Orchestration of Sight and Sound

arXiv:2608.05803v2 Announce Type: replace Abstract: Recent advances in generative video modeling have enabled diverse generation, reference-based synthesis, extension, and editing, but existing appro...

By Vorch Team, Xiaoyu Chen, Yang Ding, Cong Han, Menglin Han, Yuxin Hong, Jiebo Hou, Zequn Jie, Xiang Li, Jing Liu, Qi Liu, Yulei Lu, Siyuan Luo, Lin Ma, Xin Ma, Yinlong Qian, Peng Shi, Fang Wan, Siqi Wang, Yaohui Wang, Yaole Wang, Yidi Wu, Siqian Yang, Mingyu Yin, Haoran Yu, Gang Yue, Lisai Zhang, Yuting Zhang
Hugging Face Trending Papers
Aug 6

Vorch-Omni: Multi-Task Orchestration of Sight and Sound

Recent advances in generative video modeling have enabled diverse generation, reference-based synthesis, extension, and editing, but existing approaches often rely on fragmented task-specific models. A general model must distinguish heterogeneous target, source, and reference signals to determine what to generate, preserve, or use as guidance, while reducing interference among tasks.

arXiv Computer Vision
Oct 12

AV-FlowEdit: Training-Free Prompt-Driven Audio-Visual Editing with Event-Aware Spectral Trajectory Preservation

AV-FlowEdit is a training‑free, inversion‑free framework that enables prompt‑driven audio‑visual editing using joint audio‑video diffusion models. It leverages source‑to‑target flow differences to guide edits while preserving non‑target structure and cross‑modal correspondence. The method includes a Cross‑Modal Event‑Aware Spectral Trajectory Preservation module that suppresses drift by combining flow saliency, aligned audio activity, multiscale low‑frequency displacement, and diffusion‑stage scheduling, achieving strong prompt alignment, temporal consistency, and competitive audio quality.

By Shaohua Pan, Xiao Liu
Hugging Face Trending Papers
Jul 21

OmniReasoner: Thinking with Long Audio-Video via Native Tool Use

Long audio-video reasoning is difficult for omnimodal LLMs because the decisive evidence is often sparse, cross-modal, and too expensive to preserve with uniformly high-fidelity inputs. We introduce OmniReasoner, a tool-use post-training framework for Thinking with Long Audio-Video: omni-modal LLMs learn, via supervised fine-tuning and reinforcement learning, to decide whether and where to call a zoom-in tool before answering.