The paper introduces Pose Adaptive Dynamic FiLM Modulation for Visual Speech Recognition, addressing head‑pose variation that causes appearance changes in VSR. It proposes a Dynamic Residual FiLM (DR‑FiLM) modulator that predicts input‑dependent weights to control pose‑conditioned modulation strength. Experiments on LRS2 and LRS3 show that DR‑FiLM reduces phoneme error rates compared to unweighted multi‑pathway modulation and that deeper FiLM pathways receive higher weights as head‑pose variation increases.
By Matthew Kit Khinn Teng, Haibo Zhang, Takeshi Saitoh
arXiv:2610.19559v1 Announce Type: new
Abstract: Lipreading remains challenging due to the inherent ambiguity of visual speech cues and their sensitivity to head pose and video resolution. In this wor...
By Jing-Xuan Zhang, Genshun Wan, Jianqing Gao, Jia Pan, Yumei Zhang, Lichen Zhang
Lipreading remains challenging due to the inherent ambiguity of visual speech cues and their sensitivity to head pose and video resolution. In this work, we propose FLARE, a framework for robust visua...
arXiv:2609.20839v1 Announce Type: new
Abstract: Phoneme-centric visual speech recognition reconstructs sentences from intermediate phoneme predictions, making overall recognition performance highly d...
By Matthew Kit Khinn Teng, Haibo Zhang, Takeshi Saitoh
arXiv:2608. 05218v1 Announce Type: new Abstract: 3D Gaussian Splatting (3DGS) enables fast, photorealistic talking-head rendering, yet accurate lip articulation remains elusive: mouth motion is often over-smoothed and may violate hard articulatory constraints such as bilabial closures, producing the notorious ``leaky mouth'' artifact.
By Ao Fu, Yi Zhou
arXiv:2603.08249v2 Announce Type: replace-cross
Abstract: Audiovisual speech recognition (AVSR) combines acoustic and visual cues to improve transcription robustness under challenging conditions but...
By Pol Buitrago, Javier Hernando
PVSync is a unified model that estimates audio‑visual offset and scores phoneme‑level articulation for lip‑sync tasks. It uses window‑level contrastive learning for synchronisation and a phoneme‑level articulation objective that aligns audio and video embeddings of the same viseme class across clips. The model achieves higher correlation with human rankings and better performance on offset recovery and viseme matching benchmarks compared to existing methods.
By Kevin Stephen, Varun Menon, Timo Mertens, Nikita Drobyshev
LiveSync introduces an iterative self‑training framework for audio‑driven lip‑sync generation. The method starts with a mask‑based model trained on curated real‑video data, then uses a frozen checkpoint to generate pseudo pairs by editing mouth regions with independently sampled audio. These high‑confidence pseudo pairs are mixed with real data to refine the model, achieving state‑of‑the‑art synchronization on HDTF and strong performance on AIGC‑LipSync while keeping inference latency low.
By Huaishao Luo, Huan Zhang, Guanbo Bao, Yinglu Liu, Youzheng Wu, Xiaodong He
arXiv:2609.10394v1 Announce Type: cross
Abstract: Current audio-visual speech recognition (AVSR) benchmarks, like LRS3, rely heavily on clean, scripted and rehearsed speech. They fail to reflect the...
By Rishabh Jain, Aristeidis Papadopoulos, Zhaofeng Lin, Naomi Harte
FacEDiT is a mask‑controlled model for editing local segments of talking head videos by infilling facial motion from speech and surrounding context. It uses a self‑supervised pretext task that masks a motion span during training, enabling variable‑duration local rewriting while preserving identity, unedited regions, and boundary continuity. The authors also release FacEDiTBench, a benchmark covering diverse edit types and lengths, and demonstrate that FacEDiT achieves accurate, speech‑aligned edits with strong identity preservation and seamless transitions.
By Kim Sung-Bin, Joohyun Chang, David Harwath, Tae-Hyun Oh
Watch Your Speech (WYS) is a video‑to‑speech synthesis framework that uses textual conditioning to resolve the one‑to‑many mapping problem inherent in silent talking‑face videos. It fuses textual context with video sequences via an attention‑based embedding module and employs a conditional flow matching objective to produce high‑fidelity, phonemically accurate speech. Experiments on LRS2 and LRS3 show that WYS sets new state‑of‑the‑art results in audio‑visual synchronization while maintaining competitive word‑error rates, and subjective tests confirm near‑human naturalness.
By Gunwoo Lee, Yoori Oh, Yoseob Han
The paper introduces Cue2Narrate, a two‑stage pipeline that jointly predicts what visual events to narrate and when to insert the narration in long, untrimmed movie clips. It uses a dual‑head audio‑visual localizer to identify visual cue and narration windows, followed by a LoRA‑adapted vision‑language model that generates concise audio descriptions, trained with a Description Ranking Loss. The authors also present the LongLSMDC benchmark, comprising up to 8‑minute clips, and show that Cue2Narrate outperforms video‑only and audio‑only baselines by 5–12 points in average mAP and improves AD generation over fine‑tuned base VLMs.
By Akshita Gupta, Aditya Arora, Federico Tombari, Marcus Rohrbach, Anna Rohrbach