Lipreading remains challenging due to the inherent ambiguity of visual speech cues and their sensitivity to head pose and video resolution. In this work, we propose FLARE, a framework for robust visua...
arXiv:2606.00751v2 Announce Type: replace Abstract: Visual Speech Recognition (VSR) aims to recognize speech from visual cues such as lip movements. Still, its performance is fundamentally limited by...
arXiv:2608. 05218v1 Announce Type: new Abstract: 3D Gaussian Splatting (3DGS) enables fast, photorealistic talking-head rendering, yet accurate lip articulation remains elusive: mouth motion is often over-smoothed and may violate hard articulatory constraints such as bilabial closures, producing the notorious ``leaky mouth'' artifact.
arXiv:2607. 26742v1 Announce Type: cross Abstract: Zero-shot text-to-speech (TTS) clones a voice from a short audio prompt, but this reliance on reference audio is a barrier when only visual information is available, e.
arXiv:2610.22499v1 Announce Type: new Abstract: Human communication is inherently multimodal, conveying intent not only through linguistic and vocal cues but also through visual signals. However, exi...
arXiv:2610.19290v1 Announce Type: new Abstract: We present Kandinsky S2V 1.0, an open family of speech-to-video models for controllable audio-driven human animation, comprising Kandinsky S2V 1.0 Lite...
arXiv:2609.10394v1 Announce Type: cross Abstract: Current audio-visual speech recognition (AVSR) benchmarks, like LRS3, rely heavily on clean, scripted and rehearsed speech. They fail to reflect the...
The paper introduces AV-STE, a modular streaming audio‑visual front‑end that enhances corrupted semantic speech tokens using noisy audio and lip video before they reach a frozen speech LLM. By preserving the downstream dialogue model’s pretrained conversational abilities, AV‑STE improves response coherence from 1.42 to 1.91 in same‑dataset speaker interference scenarios while maintaining turn‑taking behavior. These gains also transfer to out‑of‑domain Seamless Interaction.
arXiv:2609.14666v1 Announce Type: cross Abstract: Turn-taking is a fundamental component of spoken interaction, and while humans naturally rely on both verbal and non-verbal signals, dialogue systems...
Vorch-Human is a unified framework for human‑centric audio‑visual generation that handles multiple tasks—animating a person from speech, jointly generating speech and video from a voice reference, and synthesizing a scene from paired appearance and voice references—using a single dual‑stream audio‑video diffusion transformer. The model incorporates clean condition‑audio and condition‑video tokens, per‑token task embeddings, temporal position types, condition masks, and a shared multimodal prompt encoder to express diverse inputs such as driving speech, timbre examples, first frames, and subject images. A two‑level data pipeline supplies the necessary supervision by extracting speech, appearance, and timbre annotations from clips and linking consistent identity and outfit references across videos, while a frozen‑prefix recurrence enables long‑form audio‑driven generation with reduced boundary discontinuity and identity drift.
PAGE (Parts-Aware Guidance) is a diffusion-based framework that improves co-speech gesture portrait video generation by enhancing local spatial fidelity and global audio‑visual consistency. It introduces Parts‑Specific Reweighting during training to focus supervision on key regions like lips, eyes, and hands, and Consistency Classifier Guidance plus Differential Guidance during inference to correct audio‑visual mismatches and refine motion separately. The authors also present the Chinese Announcer Speech (CAS) dataset, a large‑scale Chinese benchmark that expands on the PATS dataset, and demonstrate that PAGE outperforms existing methods in perceptual quality, gesture diversity, and audio‑visual alignment.
arXiv:2610.08417v1 Announce Type: new Abstract: Recent advances in LipSync generation technology have led to the creation of highly realistic videos, posing severe societal risks. However, existing d...