The paper introduces Shift-Aware Identity-guided Latent (SAIL) refinement for Referring Audio‑Visual Segmentation (Ref‑AVS), a task that requires precise segmentation of a specific object using both visual and auditory cues grounded in natural language. SAIL employs a dual‑token strategy to separate frame‑level dynamics from a global semantic anchor, a shift‑aware refinement module to correct target‑shift drift, and a decoder with continuous memory updates to preserve spatio‑temporal coherence and identity consistency. Experiments on two Ref‑AVS benchmarks show that SAIL outperforms existing AVS, Ref‑VOS, and Ref‑AVS methods.
By Kun Li, Sami Sebastian Brandt, Michael Ying Yang
arXiv:2606. 01031v1 Announce Type: cross Abstract: Audio-driven talking-head generation has advanced rapidly, yet existing evaluation protocols mainly rely on frame-wise metrics that assume strict temporal correspondence between generated and reference videos.
By Zhicheng Zhang, Lei Wang, Yu Zhang, Yongsheng Gao
BiMoGen introduces a unified masked discrete diffusion framework for bidirectional motion‑text generation, addressing the limitations of autoregressive models in capturing bidirectional dependencies between language and motion. The approach employs a two‑stage training strategy—decoupled uni‑ and cross‑modal pretraining followed by supervised fine‑tuning—to establish robust cross‑modal correspondence, and incorporates Generation‑Aware Self‑Correction to mitigate error propagation during inference. Experiments on HumanML3D and KIT‑ML show competitive performance on both text‑to‑motion and motion‑to‑text tasks, demonstrating the effectiveness of the proposed training and correction mechanisms.
Encore is a new framework for generating long, synchronized audio‑video content. It splits the problem into local continuity, handled by iterative chunk‑wise synthesis with cross‑chunk context, and global consistency, enforced through reference audio‑video signals with shifted position embeddings. The Adaptive Signal Routing mechanism learns attention biases and residual scales to modulate the influence of each conditioning signal, enabling end‑to‑end joint audio‑video generation and infinite‑length inference.
By Shaohua Pan, Junbao Chen, Shengyi He, Jingfeng Xue, Wen Tao, Haocheng Feng, Siming Fan, Dongwei Pan, Yi Yang, Wei He, Hang Zhou
BiMoGen introduces a unified masked discrete diffusion framework for bidirectional motion‑text generation, addressing limitations of autoregressive models by allowing iterative bidirectional prediction. The method employs a two‑stage training strategy—Decoupled Uni‑ and Cross‑Modal Training followed by supervised fine‑tuning—to establish cross‑modal correspondence and specialize for bidirectional generation. To mitigate train‑test mismatch, Generation‑Aware Self‑Correction exposes the model to its own predictions during training and applies correction passes early in sampling to refine unmasked tokens, achieving competitive results on HumanML3D and KIT‑ML.
By Wanjiang Weng, Yongliang Wu, Xiaofeng Tan, Xingyu Zhu, Wenbo Zhu, Hongsong Wang
arXiv:2608. 13817v1 Announce Type: cross Abstract: Human speech production is constrained by physiology, giving rise to characteristic temporal structure on acoustic signals.
By Tom\'as Andrade Weber
arXiv:2606. 18485v1 Announce Type: cross Abstract: Neural Text-to-Speech (TTS) systems achieve remarkable quality on short utterances but long-form speech generation shows prosodic drift, speaker inconsistencies and sentence boundary artifacts.
By Subhankar Ghosh, Jason Li, Paarth Neekhara, Shehzeen Hussain, Ryan Langman, Xuesong Yang, Roy Fejgin
TEMPO is a unified model that adds temporally‑grounded capabilities to large audio‑language models, enabling timestamping of events, speakers, and sounds in audio, speech, and music. It introduces a supervised fine‑tuning stage featuring atomic timestamp tokens, a time‑aware projector with sinusoidal encodings, and a distance‑aware Gaussian loss, trained via a synthetic‑to‑real curriculum. Additionally, TEMPO employs reinforcement learning (GRPO) as a refinement step, and achieves state‑of‑the‑art performance on a benchmark of 10K samples across five timestamping tasks, surpassing Audio Flamingo Next and Qwen3‑Omni.
By Apoorva Kulkarni, Kaousheik Jayakumar, Sreyan Ghosh, Utathya Aich, Ramani Duraiswami, Dinesh Manocha
Existing multi-speaker dialogue systems bind speakers to utterances through structured supervision: per-turn tags, multi-stream transcriptions, or learnable speaker embeddings. These systems operate within speech-only pipelines that produce clean vocal sequences without the ambient texture of real conversations.
arXiv:2602. 10230v2 Announce Type: replace Abstract: Audio language models process input audio into rich frame-level representations, but the standard approach to temporal localization generates timestamps as sequences of text tokens, which discards the frame-level representations in favor of autoregressive decoding.
By Joseph An, Phillip Keung, Jiaqi Wang, Orevaoghene Ahia, Noah A. Smith
arXiv:2606. 19325v1 Announce Type: cross Abstract: Existing multi-speaker dialogue systems bind speakers to utterances through structured supervision: per-turn tags, multi-stream transcriptions, or learnable speaker embeddings.
By Michael Finkelson, Daniel Segal, Eitan Richardson, Shahar Armon, Nani Goldring, Poriya Panet, Nir Zabari, Benjamin Brazowski, Or Patashnik, Yoav HaCohen
Motion-Omni is an end‑to‑end framework that jointly generates spoken dialogue and full‑body motion, producing speech, facial expressions, and hand, upper‑body, and lower‑body movements directly from the hidden states of a language model. The system requires joint training of the language model, speech generator, and motion generator to maintain audio‑motion alignment, and it is supervised using a scalable, model‑agnostic pipeline that pseudo‑labels 422,856 speech‑motion pairs. With a Qwen2.5‑7B‑Instruct backbone, Motion‑Omni‑Q7 achieves near‑cascade performance on motion metrics while being 5.4× faster, and it outperforms other non‑teacher cascades on beat correlation, diversity, and word error rate.
By Chengqian Ma, Wei Tao, Haoyu Zhang, Yiwen Guo