← Back to all news
arXiv Computer Vision By Kevin Stephen, Varun Menon, Timo Mertens, Nikita Drobyshev

PVSync: A Unified Lip-Sync Expert for Timing and Articulation

Read the original on arXiv Computer Vision →

PVSync is a unified model that estimates audio‑visual offset and scores phoneme‑level articulation for lip‑sync tasks. It uses window‑level contrastive learning for synchronisation and a phoneme‑level articulation objective that aligns audio and video embeddings of the same viseme class across clips. The model achieves higher correlation with human rankings and better performance on offset recovery and viseme matching benchmarks compared to existing methods.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computer Vision.

Related stories

arXiv Computer Vision
Oct 2

Align Then Reason: A Multimodal Lip-Sync Judge for Dubbing

The paper introduces Align Then Reason (ATR), a multilingual lip‑sync judge that aligns frame‑level lip representations with phonetic units of a candidate text line and then uses a language model to evaluate both content and timing. ATR achieves significant improvements over existing baselines on a seven‑language benchmark, with mean AUC gains of up to 59.4% for 2B reasoners and similar gains across other LLM families. The method also transfers well to unseen languages and outperforms lip‑reading baselines on real dubbing tasks such as dub‑line reranking and script‑to‑clip assignment.

By Rui Liu, Bhavin Jawade, Haoqi Li, Shivam Mehta, Karan Saxena, Yinghong Lan, Cameron R. Wolfe
llmsragfine-tuningmultimodalbenchmarkssafety
More like this →
arXiv Computer Vision
Oct 12

Mechanisms of Multimodal Synchronization: Insights from Decoder-Based Video-Text-to-Speech Synthesis

The paper investigates how sequence design influences temporal synchronization in unified decoder-only transformers for video‑text‑to‑speech synthesis. By training a Visatronic model on VoxCeleb2, the authors compare positional encoding schemes and modality ordering, finding that global sequential indexing and co‑temporal ordered indexing yield similar in‑domain performance, while text‑first co‑temporal indexing best transfers under domain shift. They also introduce TimeSync, a phoneme‑level metric that uncovers misalignments missed by frame‑level measures, and demonstrate that text and video provide complementary cues for intelligibility and temporal/emotional expressiveness.

By Akshita Gupta, Tatiana Likhomanenko, Karren Dai Yang, Richard He Bai, Zakaria Aldeneh, Navdeep Jaitly
llmsmultimodalsafety
More like this →