UniAVC is a unified audio‑video model that supports text‑conditioned joint synthesis, bidirectional cross‑modal generation, and instruction‑guided editing for both modalities. It first trains separate audio and video branches on large unimodal datasets, then fine‑tunes with paired data to learn cross‑modal relationships. The model uses modality‑decoupled noise schedules and asymmetric training to enable tasks such as intent‑aware editing and post‑inference self‑refinement without task‑specific adapters.
arXiv:2604.15086v3 Announce Type: replace-cross Abstract: Recent advances in video-to-audio (V2A) generation enable high-quality audio synthesis from visual content, yet achieving robust and fine-gra...
arXiv:2608.31106v1 Announce Type: new Abstract: Recent video generators often omit audio or synthesize it in a separate stage, limiting reciprocal modeling of visual dynamics and acoustic events. We...
The paper investigates how sequence design influences temporal synchronization in unified decoder-only transformers for video‑text‑to‑speech synthesis. By training a Visatronic model on VoxCeleb2, the authors compare positional encoding schemes and modality ordering, finding that global sequential indexing and co‑temporal ordered indexing yield similar in‑domain performance, while text‑first co‑temporal indexing best transfers under domain shift. They also introduce TimeSync, a phoneme‑level metric that uncovers misalignments missed by frame‑level measures, and demonstrate that text and video provide complementary cues for intelligibility and temporal/emotional expressiveness.
Recent advances in generative video modeling have enabled diverse generation, reference-based synthesis, extension, and editing, but existing approaches often rely on fragmented task-specific models. A general model must distinguish heterogeneous target, source, and reference signals to determine what to generate, preserve, or use as guidance, while reducing interference among tasks.
arXiv:2610.00691v1 Announce Type: new Abstract: Recent joint audio-video generative models can synthesize realistic videos with synchronized sound, but typically generate audio as a single mixed trac...
The paper introduces Modality Self-Play (MSP), a training framework that enables multimodal large language models to learn cross‑modal interactions without requiring paired non‑text data. MSP uses proxy‑mediated textual descriptions of unseen modalities as semantic bridges, allowing a model to generate and align these proxies with observed modalities. Experiments show that MSP achieves state‑of‑the‑art or competitive performance on several multimodal benchmarks and demonstrates effective zero‑shot modality composition through learned bridges and anchor‑supporting proxy descriptions.
arXiv:2607.18227v2 Announce Type: replace Abstract: In line with the prevailing direction of vision research, we explore the integration of both generation and editing capabilities for video and imag...
arXiv:2608.05803v2 Announce Type: replace Abstract: Recent advances in generative video modeling have enabled diverse generation, reference-based synthesis, extension, and editing, but existing appro...
arXiv:2609.08275v1 Announce Type: new Abstract: Recent multi-shot audio-video generators can produce increasingly coherent and cinematic outputs, but coherence does not imply the ability to execute e...
AV-FlowEdit is a training‑free, inversion‑free framework that enables prompt‑driven audio‑visual editing using joint audio‑video diffusion models. It leverages source‑to‑target flow differences to guide edits while preserving non‑target structure and cross‑modal correspondence. The method includes a Cross‑Modal Event‑Aware Spectral Trajectory Preservation module that suppresses drift by combining flow saliency, aligned audio activity, multiscale low‑frequency displacement, and diffusion‑stage scheduling, achieving strong prompt alignment, temporal consistency, and competitive audio quality.
arXiv:2610.27947v1 Announce Type: new Abstract: Human-centric audio-visual editing requires precise alignment of lip movements, gestures, and expressions with speech. Existing methods process inputs...
TimeSteer introduces inference‑time speech scheduling for joint audio‑visual diffusion models, enabling users to place speech and visual articulation within specified time intervals without fine‑tuning the backbone. The method leverages two properties of the denoising process: a timing‑sensitive text‑to‑audio cross‑attention head that reveals each utterance’s source span, and a predicted clean latent that already organizes coupled speech and visual content. TimeSteer localizes each utterance’s source span and remaps the associated audio‑visual latent to the target interval, and the authors present SpeechShift as the first benchmark for interval‑level speech scheduling.