TimeSteer introduces inference‑time speech scheduling for joint audio‑visual diffusion models, enabling users to place speech and visual articulation within specified time intervals without fine‑tuning the backbone. The method leverages two properties of the denoising process: a timing‑sensitive text‑to‑audio cross‑attention head that reveals each utterance’s source span, and a predicted clean latent that already organizes coupled speech and visual content. TimeSteer localizes each utterance’s source span and remaps the associated audio‑visual latent to the target interval, and the authors present SpeechShift as the first benchmark for interval‑level speech scheduling.
arXiv:2610.32042v1 Announce Type: new Abstract: Cross-lingual dubbing aims to convert source speech into target-language speech that remains synchronized with the original video. The key to this task...
arXiv:2608.05663v3 Announce Type: replace Abstract: Real-time long-form avatar audio-video generation requires causal, continuous synthesis while maintaining audiovisual synchronization and visual co...
Real-time long-form avatar audio--video generation requires causal, continuous synthesis while maintaining audiovisual synchronization and visual consistency. Adapting a pretrained bidirectional model to this setting presents two key dilemmas.
TEMPO is a unified model that adds temporally‑grounded capabilities to large audio‑language models, enabling timestamping of events, speakers, and sounds in audio, speech, and music. It introduces a supervised fine‑tuning stage featuring atomic timestamp tokens, a time‑aware projector with sinusoidal encodings, and a distance‑aware Gaussian loss, trained via a synthetic‑to‑real curriculum. Additionally, TEMPO employs reinforcement learning (GRPO) as a refinement step, and achieves state‑of‑the‑art performance on a benchmark of 10K samples across five timestamping tasks, surpassing Audio Flamingo Next and Qwen3‑Omni.
UniAVC is a unified audio‑video model that supports text‑conditioned joint synthesis, bidirectional cross‑modal generation, and instruction‑guided editing for both modalities. It first trains separate audio and video branches on large unimodal datasets, then fine‑tunes with paired data to learn cross‑modal relationships. The model uses modality‑decoupled noise schedules and asymmetric training to enable tasks such as intent‑aware editing and post‑inference self‑refinement without task‑specific adapters.
arXiv:2604.15086v3 Announce Type: replace-cross Abstract: Recent advances in video-to-audio (V2A) generation enable high-quality audio synthesis from visual content, yet achieving robust and fine-gra...
Recent advances in generative video modeling have enabled diverse generation, reference-based synthesis, extension, and editing, but existing approaches often rely on fragmented task-specific models. A general model must distinguish heterogeneous target, source, and reference signals to determine what to generate, preserve, or use as guidance, while reducing interference among tasks.
The paper investigates how sequence design influences temporal synchronization in unified decoder-only transformers for video‑text‑to‑speech synthesis. By training a Visatronic model on VoxCeleb2, the authors compare positional encoding schemes and modality ordering, finding that global sequential indexing and co‑temporal ordered indexing yield similar in‑domain performance, while text‑first co‑temporal indexing best transfers under domain shift. They also introduce TimeSync, a phoneme‑level metric that uncovers misalignments missed by frame‑level measures, and demonstrate that text and video provide complementary cues for intelligibility and temporal/emotional expressiveness.
arXiv:2608.05803v2 Announce Type: replace Abstract: Recent advances in generative video modeling have enabled diverse generation, reference-based synthesis, extension, and editing, but existing appro...
arXiv:2606. 01031v1 Announce Type: cross Abstract: Audio-driven talking-head generation has advanced rapidly, yet existing evaluation protocols mainly rely on frame-wise metrics that assume strict temporal correspondence between generated and reference videos.
arXiv:2608. 04902v1 Announce Type: cross Abstract: Video-to-audio (V2A) generation extends image-to-audio generation (I2A) by introducing consecutive frames that provide essential temporal cues for audio synthesis.
Vorch-Human is a unified framework for human‑centric audio‑visual generation that handles multiple tasks—animating a person from speech, jointly generating speech and video from a voice reference, and synthesizing a scene from paired appearance and voice references—using a single dual‑stream audio‑video diffusion transformer. The model incorporates clean condition‑audio and condition‑video tokens, per‑token task embeddings, temporal position types, condition masks, and a shared multimodal prompt encoder to express diverse inputs such as driving speech, timbre examples, first frames, and subject images. A two‑level data pipeline supplies the necessary supervision by extracting speech, appearance, and timbre annotations from clips and linking consistent identity and outfit references across videos, while a frozen‑prefix recurrence enables long‑form audio‑driven generation with reduced boundary discontinuity and identity drift.