arXiv Computer Vision September 25, 2026 By Matthew Kit Khinn Teng, Haibo Zhang, Takeshi Saitoh

Pose Adaptive Dynamic FiLM Modulation for Visual Speech Recognition

Read the original on arXiv Computer Vision →

The paper introduces Pose Adaptive Dynamic FiLM Modulation for Visual Speech Recognition, addressing head‑pose variation that causes appearance changes in VSR. It proposes a Dynamic Residual FiLM (DR‑FiLM) modulator that predicts input‑dependent weights to control pose‑conditioned modulation strength. Experiments on LRS2 and LRS3 show that DR‑FiLM reduces phoneme error rates compared to unweighted multi‑pathway modulation and that deeper FiLM pathways receive higher weights as head‑pose variation increases.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computer Vision.

multimodal

arXiv Computer Vision

Sep 15

Head-Pose-Aware Visual Speech Recognition with FiLM Modulation

arXiv:2606.00751v2 Announce Type: replace Abstract: Visual Speech Recognition (VSR) aims to recognize speech from visual cues such as lip movements. Still, its performance is fundamentally limited by...

By Matthew Kit Khinn Teng, Haibo Zhang, Takeshi Saitoh

llms multimodal benchmarks

arXiv AI

Aug 3

DoubleHelix: Structured Cross-Modal Fusion for Audio-Visual Speech Recognition with LLMs

arXiv:2607. 29112v1 Announce Type: cross Abstract: Audio-visual speech recognition (AVSR) relies on effective fusion of audio and visual modalities, yet existing approaches treat cross-modal interaction as a single-step operation without structured iterative refinement.

By Ziwei Cheng, Zhenhua Tan, Zhuomin Zhu

llms multimodal benchmarks safety

arXiv Computation and Language

Sep 21

Curriculum-Based Noise Adaptation for Phoneme-to-Text Reconstruction in Visual Speech Recognition

arXiv:2609.20839v1 Announce Type: new Abstract: Phoneme-centric visual speech recognition reconstructs sentences from intermediate phoneme predictions, making overall recognition performance highly d...

By Matthew Kit Khinn Teng, Haibo Zhang, Takeshi Saitoh

multimodal benchmarks

arXiv Computer Vision

Sep 10

AVSRBench: A Multi-Condition AVSR Benchmark

arXiv:2609.10366v1 Announce Type: cross Abstract: While AVSR has achieved sub-1% word error rates on the standard LRS3 benchmark, its reliance on broadcast speech obscures whether this reflects true...

By Rishabh Jain, Naomi Harte

llms fine-tuning multimodal benchmarks

arXiv Computation and Language

Sep 30

Bootstrapping Audiovisual Speech Recognition in Zero-AV-Resource Scenarios

arXiv:2603.08249v2 Announce Type: replace-cross Abstract: Audiovisual speech recognition (AVSR) combines acoustic and visual cues to improve transcription robustness under challenging conditions but...

By Pol Buitrago, Javier Hernando

fine-tuning multimodal benchmarks

arXiv AI

Sep 17

GrainSpeech: Less Context, More Detail for Compact Speech Synthesis

GrainSpeech is a compact speech synthesis model that uses a fixed‑receptive‑field convolutional encoder to reduce pitch, energy, and duration prediction errors by 36.0%, 17.3%, and 3.4% respectively. It introduces a Mel‑specific gradient‑variance supervision that improves fine‑scale variation while avoiding quality degradation. With only 264.8K parameters, GrainSpeech achieves 17.9× real‑time Mel generation on a microcontroller and attains UTMOS scores comparable to much larger models, using less than 1.5% of their parameters.

By Zitao Liang, Chang Gao