← Back to all news
arXiv AI By Chao Zhou, Yiling Chen, Qi Chu, Tao Gong, Nenghai Yu, Tianyi We

TimeSteer: Inference-Time Speech Scheduling in Joint Audio-Visual Diffusion Models

Read the original on arXiv AI →

TimeSteer introduces inference‑time speech scheduling for joint audio‑visual diffusion models, enabling users to place speech and visual articulation within specified time intervals without fine‑tuning the backbone. The method leverages two properties of the denoising process: a timing‑sensitive text‑to‑audio cross‑attention head that reveals each utterance’s source span, and a predicted clean latent that already organizes coupled speech and visual content. TimeSteer localizes each utterance’s source span and remaps the associated audio‑visual latent to the target interval, and the authors present SpeechShift as the first benchmark for interval‑level speech scheduling.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Sep 1

TEMPO: Temporally-grounded Multi-task Post-training for Large Audio-Language Models

TEMPO is a unified model that adds temporally‑grounded capabilities to large audio‑language models, enabling timestamping of events, speakers, and sounds in audio, speech, and music. It introduces a supervised fine‑tuning stage featuring atomic timestamp tokens, a time‑aware projector with sinusoidal encodings, and a distance‑aware Gaussian loss, trained via a synthetic‑to‑real curriculum. Additionally, TEMPO employs reinforcement learning (GRPO) as a refinement step, and achieves state‑of‑the‑art performance on a benchmark of 10K samples across five timestamping tasks, surpassing Audio Flamingo Next and Qwen3‑Omni.

By Apoorva Kulkarni, Kaousheik Jayakumar, Sreyan Ghosh, Utathya Aich, Ramani Duraiswami, Dinesh Manocha
llmsragreinforcement-learningfine-tuningmultimodalbenchmarks
More like this →
arXiv Computer Vision
Oct 21

UniAVC: Unified Audio-Video Creation via Progressive Training and Asymmetric Scheduling

UniAVC is a unified audio‑video model that supports text‑conditioned joint synthesis, bidirectional cross‑modal generation, and instruction‑guided editing for both modalities. It first trains separate audio and video branches on large unimodal datasets, then fine‑tunes with paired data to learn cross‑modal relationships. The model uses modality‑decoupled noise schedules and asymmetric training to enable tasks such as intent‑aware editing and post‑inference self‑refinement without task‑specific adapters.

By Kien T. Pham, Yuhao Liu, Chong Hou Choi, Zhenqi He, Yanghao Wang, Chenyang Lei, Rui Liu, Qifeng Chen, Long Chen
multimodal
More like this →
arXiv Computer Vision
Sep 30

ControlFoley: Unified and Controllable Video-to-Audio Generation with Cross-Modal Conflict Handling

arXiv:2604.15086v3 Announce Type: replace-cross Abstract: Recent advances in video-to-audio (V2A) generation enable high-quality audio synthesis from visual content, yet achieving robust and fine-gra...

By Jianxuan Yang, Xinyue Guo, Zhi Cheng, Kai Wang, Lipan Zhang, Jinjie Hu, Qiang Ji, Yihua Cao, Yihao Meng, Zhaoyue Cui, Mengmei Liu, Meng Meng, Jian Luan
multimodalbenchmarkssafety
More like this →