arXiv AI September 16, 2026 By Kaidi Wang, Yi He, Wenhao Guan, Weijie Wu, Peijie Chen, Hongwu Ding, Xiong Zhang, Di Wu, Meng Meng, Jian Luan, Lin Li, Qingyang Hong

SyncVoice: Simple and Effective Automatic Video Dubbing with Vision-Augmented TTS

Read the original on arXiv AI →

SyncVoice is a new automatic video dubbing framework that adds a lightweight Text‑Visual Fusion Module to a pretrained TTS system, aligning visual features with linguistic representations to produce temporally synchronized speech. The approach avoids complex architectural changes and achieves state‑of‑the‑art performance on the LRS3 dataset in zero‑shot dubbing. When further trained on a large bilingual audio‑visual corpus, SyncVoice improves vocal fidelity while maintaining synchronization, enabling a single model to dub both Chinese and English videos.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv Computer Vision

Sep 10

AVSRBench: A Multi-Condition AVSR Benchmark

arXiv:2609.10366v1 Announce Type: cross Abstract: While AVSR has achieved sub-1% word error rates on the standard LRS3 benchmark, its reliance on broadcast speech obscures whether this reflects true...

By Rishabh Jain, Naomi Harte

llms fine-tuning multimodal benchmarks

arXiv Computation and Language

Sep 4

Alignment-Free Text-Audiobox for Voice Dubbing and Full-Duplex Dialogue Synthesis

The paper introduces Alignment-Free Text‑Audiobox (Text‑AB), a unified diffusion‑based framework that performs high‑quality voice dubbing and full‑duplex dialogue synthesis without requiring forced alignment. Text‑AB uses a latent diffusion model with DAC‑VAE features, achieving over 10× compression compared to prior EnCodec representations, and learns text‑speech alignment via cross‑attention. The authors pretrain a 3B‑parameter model on 480k hours of monolingual speech and fine‑tune it for cross‑lingual dubbing, full‑duplex dialogue, and emotional dialogue, reporting significant improvements in prosody, voice similarity, naturalness, and emotional expressivity over existing internal systems.

By Sanyuan Chen, Min-Jae Hwang, Sho Inoue, Anna Sun, Bokai Yu, David Kant, Dongmin Hyun, Dorian Desblancs, Gregory Antonovsky, Oleg Repin, Peng-Jen Chen, Xutai Ma, Zehai Tu, Juan Pino, Wei-Ning Hsu

llms rag diffusion fine-tuning benchmarks safety

arXiv AI

Jun 16

Pixel-TTS: Image based Text Rendering for Robust Text-to-Speech

arXiv:2606. 14750v1 Announce Type: cross Abstract: Recent advances in pixel-based text modeling show that representing text as images enables models to exploit visual cues for language understanding.

By Adarsh Arigala, Arjun Gangwar, S Umesh, Yova Kementchedjhieva

rag fine-tuning multimodal

arXiv AI

Jun 2

Temporally-Aligned Evaluation for Audio-Driven Talking Head Generation

arXiv:2606. 01031v1 Announce Type: cross Abstract: Audio-driven talking-head generation has advanced rapidly, yet existing evaluation protocols mainly rely on frame-wise metrics that assume strict temporal correspondence between generated and reference videos.

By Zhicheng Zhang, Lei Wang, Yu Zhang, Yongsheng Gao

benchmarks safety

arXiv Machine Learning

Aug 13

CookVoice: Unified Framework for Style Controllable Multi-Modal Human Voice Generation

arXiv:2608. 11590v1 Announce Type: cross Abstract: Human voice generation has made rapid progress in speech generation, singing voice generation, voice cloning, and voice editing.

By Haowei Lou, Hye-Young Paik, Dai Jia, Kai Li, Lina Yao

multimodal safety

Hugging Face Trending Papers

Aug 6

Vorch-Streamer: Extending Human Audio-Visual Generation to Real-Time Long-Form Streaming

Real-time long-form avatar audio--video generation requires causal, continuous synthesis while maintaining audiovisual synchronization and visual consistency. Adapting a pretrained bidirectional model to this setting presents two key dilemmas.

llms diffusion efficiency multimodal safety