The paper introduces Emo-DVS, a large-scale, multimodal dataset combining event camera, audio, and text data for emotion recognition, designed to mitigate privacy concerns associated with RGB cameras. It proposes the Information‑Guided Gated Fusion (IGF) framework, which pre‑trains an event encoder on the dataset’s FAU subset, adaptively gates modalities to reduce noise, and aligns cross‑modal representations via mutual information maximization. Experiments show that IGF outperforms existing methods on this challenging tri‑modal benchmark.
By Jiaqi Chen, Qinfu Xu, Hao Zhuang, Liyuan Pan
arXiv:2608. 15999v1 Announce Type: new Abstract: Automatic emotion assessment can benefit from combining neural and behavioral signals, but many multimodal approaches rely on separate, modality-specific feature-extraction pipelines before fusion.
By Stefanos Gkikas, Eric Nichols, Christian Arzate Cruz, Randy Gomez
The paper presents emg2face, a system that uses high‑density surface electromyography (HD‑sEMG) to capture facial expressions without optical cameras, addressing issues of occlusion and privacy. It records 64 EMG channels with textile grids, synchronizes the data with video using analog audio bursts, and fits a high‑resolution parametric head model to 3D facial landmarks. A deep neural network then predicts blendshape parameters from the EMG signals, enabling real‑time facial animation on various characters.
By Ganidhu Abey, Wendy Greening, Ashika Kamboj, Leonhard Helminger, Abhijeet Ghosh, Karel Petranek, Sergio Orts Escolano, Dinesh K. Pai
Conventional face recognition relies on static appearance cues and degrades in unconstrained settings with expression variation, occlusion, and poor lighting. We hypothesize that audiovisual expression dynamics carry identity-discriminative information complementary to static appearance, and that extracting this signal requires multimodal representations robust to the variable input quality of in-the-wild video.
Facial movements convey subtle and important information that is critical for human social communication. Optical methods for face capture are difficult or impossible to use when the face is occluded...
arXiv:2606. 00170v1 Announce Type: cross Abstract: In recent years, emotion recognition based on physiological signals such as electroencephalogram (EEG) has gained considerable attention, as internal physiological data offer greater objectivity and reliability compared to external behavioral data like facial expressions.
By Zheng Wang, Shuo Wang, Junhong Wang
arXiv:2609.13255v1 Announce Type: new
Abstract: Facial state analysis plays a crucial role in understanding human expressions, psychological modeling, and human computer interaction. Traditional unim...
By Xuri Ge, Tianshuo Zhang, Ruihan Li, Hui Ye, Kaiwen Zheng, Junchen Fu, Da Huo, Joemon M. Jose, Hu Han
The paper introduces a multimodal emotion recognition framework that combines audio and visual feature extraction with an attention-based fusion strategy. Audio features include Wav2Vec2 embeddings, MFCCs, and statistical acoustic descriptors, fused via a BiLSTM, while video features are extracted using a ResNet50-BiLSTM architecture. A multi-head attention mechanism fuses these modalities, and experiments on MELD and IEMOCAP show significant accuracy and robustness gains, especially in unbalanced data settings.
By Xu Lin, Ke Wang, Hui Kang, Xinying Wang
arXiv:2607. 00410v1 Announce Type: cross Abstract: Recent brain decoding studies have made substantial progress in reconstructing externally perceived visual content from neural signals.
By Zhenhang Li, Xin Zhou, Hao Deng, Lijun Yin
arXiv:2609.12668v1 Announce Type: new
Abstract: Recent deepfake detection studies increasingly suggest remote photoplethysmography (rPPG) signals as an authenticity cue. However, existing benchmarks...
By Chenxi Yang, Yassine Ouzar, Larbi Boubchir
The paper introduces a multimodal emotion recognition system that integrates facial expressions, speech, text, and body movement to aid clinicians in psychological evaluation. While individual modalities perform well on benchmarks, their real‑world accuracy drops to 82.97%–87.64%, whereas the fused multimodal system reaches 96.43% accuracy with no state below 92.31%. These findings demonstrate that multimodal fusion enhances robustness in realistic settings, supporting more reliable evaluator assistance.
By Kris Kraack
arXiv:2608. 08873v1 Announce Type: cross Abstract: Facial Emotion Recognition (FER) is an important task that has significant implications across various fields such as biometrics, health, and human-computer interaction.
By Aya Manel Zitouni, Aicha Zenakhri, Karim Haroun, Larbi Boubchir