arXiv:2610.32042v1 Announce Type: new Abstract: Cross-lingual dubbing aims to convert source speech into target-language speech that remains synchronized with the original video. The key to this task...
TEMPO is a unified model that adds temporally‑grounded capabilities to large audio‑language models, enabling timestamping of events, speakers, and sounds in audio, speech, and music. It introduces a supervised fine‑tuning stage featuring atomic timestamp tokens, a time‑aware projector with sinusoidal encodings, and a distance‑aware Gaussian loss, trained via a synthetic‑to‑real curriculum. Additionally, TEMPO employs reinforcement learning (GRPO) as a refinement step, and achieves state‑of‑the‑art performance on a benchmark of 10K samples across five timestamping tasks, surpassing Audio Flamingo Next and Qwen3‑Omni.
The paper introduces a training-data construction strategy that combines professional dubbing with multilingual voice conversion to provide holistic parallel supervision for expressive speech-to-speech translation (S2ST). This approach aligns source and target speech in content, prosody, timing, and speaker identity, addressing limitations of existing datasets. Experiments on English‑German and English‑Spanish demonstrate that this supervision improves expressive realization across evaluated systems, with the proposed DirectS2ST method achieving the largest gains.
arXiv:2608. 02673v1 Announce Type: cross Abstract: Speech editing for content creation requires precise control over both what an edit should do and where it should apply.
The paper introduces FAST-CAP, a causality‑aware framework for simultaneous speech‑to‑speech translation that combines a factorized S2ST architecture, an adaptive policy, and a new latency metric. It employs a novel data pipeline to generate high‑fidelity, causally aligned segments, improving voice transfer and reducing the need for large training datasets. Experiments on Spanish, German, and French demonstrate that FAST‑CAP outperforms fixed‑policy baselines, achieving up to +1.2 BLEU, 26% lower latency, and a 38.8% relative latency reduction while maintaining speaker fidelity.
arXiv:2609.38658v1 Announce Type: cross Abstract: TTS systems with autoregressive semantic modeling have demonstrated strong zero-shot voice cloning performance and rich expressive variation, but the...
X-Translator is a low‑cost, modular real‑time speech‑to‑speech translation system that integrates streaming ASR, machine translation, and prompt‑conditioned TTS, managed by a session‑level runtime controller. It uses incremental segment commitment to stabilize ASR streams and an online speaker prompt manager to maintain speaker consistency across multi‑speaker conversations. The system is evaluated on translation quality, speech naturalness, latency, and speaker preservation using OpenSTBench, and its code and demo are publicly available on GitHub.
arXiv:2607.05364v4 Announce Type: replace-cross Abstract: Modern autoregressive ASR systems can emit timestamps as decoded tokens, enabling timestamped transcription without frame-level aligners or i...
arXiv:2609.22697v1 Announce Type: new Abstract: Recently, text-to-speech systems have made significant progress in speech expressiveness and controllability. However, the speaking style of generated...
arXiv:2607. 05364v1 Announce Type: cross Abstract: Modern autoregressive ASR systems can emit timestamps as decoded tokens, enabling timestamped transcription without frame-level aligners or inference-time post-processing.
arXiv:2606. 01031v1 Announce Type: cross Abstract: Audio-driven talking-head generation has advanced rapidly, yet existing evaluation protocols mainly rely on frame-wise metrics that assume strict temporal correspondence between generated and reference videos.
Modern autoregressive ASR systems can emit timestamps as decoded tokens, enabling timestamped transcription without frame-level aligners or inference-time post-processing. We show that these generated timestamps can drift across long non-speech spans: the transcript may remain plausible, but the decoded time axis drifts away from the audio.