arXiv Computer Vision
Vorch-Omni: Multi-Task Orchestration of Sight and Sound
Read the original on arXiv Computer Vision →The Flow has not summarised this story yet — read it at arXiv Computer Vision.
Related stories
Hugging Face Trending Papers
Aug 6
Vorch-Omni: Multi-Task Orchestration of Sight and Sound
Recent advances in generative video modeling have enabled diverse generation, reference-based synthesis, extension, and editing, but existing approaches often rely on fragmented task-specific models. A general model must distinguish heterogeneous target, source, and reference signals to determine what to generate, preserve, or use as guidance, while reducing interference among tasks.
arXiv Computer Vision
Oct 2
Soundwich: Video Generation with Layered and Controllable Audio
arXiv:2610.00691v1 Announce Type: new Abstract: Recent joint audio-video generative models can synthesize realistic videos with synchronized sound, but typically generate audio as a single mixed trac...
arXiv AI
Jul 24
OmniCustom: Sync Audio-Video Customization Via Joint Audio-Video Generation Model
arXiv:2602. 12304v5 Announce Type: replace-cross Abstract: Existing mainstream video customization methods focus on generating identity-consistent videos based on given reference images and textual prompts.
arXiv Computer Vision
Sep 1
DreamX-Creator: Democratizing Native Audio-Video Generation at 2K Resolution
arXiv:2608.31106v1 Announce Type: new Abstract: Recent video generators often omit audio or synthesize it in a separate stage, limiting reciprocal modeling of visual dynamics and acoustic events. We...
arXiv Computer Vision
Sep 23
Vorch-Human: Unified Multi-Task Human-Centric Generation via Long-Horizon Continuation
Vorch-Human is a unified framework for human‑centric audio‑visual generation that handles multiple tasks—animating a person from speech, jointly generating speech and video from a voice reference, and synthesizing a scene from paired appearance and voice references—using a single dual‑stream audio‑video diffusion transformer. The model incorporates clean condition‑audio and condition‑video tokens, per‑token task embeddings, temporal position types, condition masks, and a shared multimodal prompt encoder to express diverse inputs such as driving speech, timbre examples, first frames, and subject images. A two‑level data pipeline supplies the necessary supervision by extracting speech, appearance, and timbre annotations from clips and linking consistent identity and outfit references across videos, while a frozen‑prefix recurrence enables long‑form audio‑driven generation with reduced boundary discontinuity and identity drift.
arXiv Machine Learning
Aug 6
Visual Representation Matters: Exploiting Temporal Differences in Video-to-Audio Generation
arXiv:2608. 04902v1 Announce Type: cross Abstract: Video-to-audio (V2A) generation extends image-to-audio generation (I2A) by introducing consecutive frames that provide essential temporal cues for audio synthesis.