arXiv Computer Vision
Align Then Reason: A Multimodal Lip-Sync Judge for Dubbing
Read the original on arXiv Computer Vision →The Flow has not summarised this story yet — read it at arXiv Computer Vision.
Related stories
arXiv AI
Jul 9
Gradient-Based Speech-to-Text Alignment for Any ASR Model: From CTC to Speech LLMs
arXiv:2607. 06831v1 Announce Type: cross Abstract: Speech-to-text alignment means finding the temporal boundaries of each word in the audio.
arXiv Computation and Language
Sep 4
Alignment-Free Text-Audiobox for Voice Dubbing and Full-Duplex Dialogue Synthesis
The paper introduces Alignment-Free Text‑Audiobox (Text‑AB), a unified diffusion‑based framework that performs high‑quality voice dubbing and full‑duplex dialogue synthesis without requiring forced alignment. Text‑AB uses a latent diffusion model with DAC‑VAE features, achieving over 10× compression compared to prior EnCodec representations, and learns text‑speech alignment via cross‑attention. The authors pretrain a 3B‑parameter model on 480k hours of monolingual speech and fine‑tune it for cross‑lingual dubbing, full‑duplex dialogue, and emotional dialogue, reporting significant improvements in prosody, voice similarity, naturalness, and emotional expressivity over existing internal systems.
arXiv Computation and Language
Sep 10
StreamAlign: Streaming Text-Aligned Speech Tokenization
arXiv:2609.09719v1 Announce Type: new Abstract: Text-aligned speech tokenization methods have emerged to better align speech tokens with LLM token spaces, enabling more effective utilization of pretr...
arXiv AI
Jun 2
Temporally-Aligned Evaluation for Audio-Driven Talking Head Generation
arXiv:2606. 01031v1 Announce Type: cross Abstract: Audio-driven talking-head generation has advanced rapidly, yet existing evaluation protocols mainly rely on frame-wise metrics that assume strict temporal correspondence between generated and reference videos.
arXiv AI
Jun 3
AnyAudio-Judge: A Dynamic Rubric-Based Benchmark and Evaluator for Audio Instruction Following
arXiv:2606. 03116v1 Announce Type: cross Abstract: The rapid advancement of instruction-guided audio generation has highlighted the critical need for robust alignment evaluation.
Hugging Face Trending Papers
Aug 12
Confucius4-TTS: Transcript-Free Cross-Lingual Zero-Shot TTS with a Learnable Speaker Encoder
Recent advances in zero-shot text-to-speech (TTS) have substantially improved speech quality and voice cloning fidelity. However, many zero-shot TTS systems still depend on audio prompt transcripts at inference time.