The paper introduces Emo-DVS, a large-scale, multimodal dataset combining event camera, audio, and text data for emotion recognition, designed to mitigate privacy concerns associated with RGB cameras. It proposes the Information‑Guided Gated Fusion (IGF) framework, which pre‑trains an event encoder on the dataset’s FAU subset, adaptively gates modalities to reduce noise, and aligns cross‑modal representations via mutual information maximization. Experiments show that IGF outperforms existing methods on this challenging tri‑modal benchmark.
By Jiaqi Chen, Qinfu Xu, Hao Zhuang, Liyuan Pan
arXiv:2608. 15999v1 Announce Type: new Abstract: Automatic emotion assessment can benefit from combining neural and behavioral signals, but many multimodal approaches rely on separate, modality-specific feature-extraction pipelines before fusion.
By Stefanos Gkikas, Eric Nichols, Christian Arzate Cruz, Randy Gomez
Conventional face recognition relies on static appearance cues and degrades in unconstrained settings with expression variation, occlusion, and poor lighting. We hypothesize that audiovisual expression dynamics carry identity-discriminative information complementary to static appearance, and that extracting this signal requires multimodal representations robust to the variable input quality of in-the-wild video.
arXiv:2606. 00170v1 Announce Type: cross Abstract: In recent years, emotion recognition based on physiological signals such as electroencephalogram (EEG) has gained considerable attention, as internal physiological data offer greater objectivity and reliability compared to external behavioral data like facial expressions.
By Zheng Wang, Shuo Wang, Junhong Wang
arXiv:2609.13255v1 Announce Type: new
Abstract: Facial state analysis plays a crucial role in understanding human expressions, psychological modeling, and human computer interaction. Traditional unim...
By Xuri Ge, Tianshuo Zhang, Ruihan Li, Hui Ye, Kaiwen Zheng, Junchen Fu, Da Huo, Joemon M. Jose, Hu Han
The paper introduces a multimodal emotion recognition framework that combines audio and visual feature extraction with an attention-based fusion strategy. Audio features include Wav2Vec2 embeddings, MFCCs, and statistical acoustic descriptors, fused via a BiLSTM, while video features are extracted using a ResNet50-BiLSTM architecture. A multi-head attention mechanism fuses these modalities, and experiments on MELD and IEMOCAP show significant accuracy and robustness gains, especially in unbalanced data settings.
By Xu Lin, Ke Wang, Hui Kang, Xinying Wang
arXiv:2607. 00410v1 Announce Type: cross Abstract: Recent brain decoding studies have made substantial progress in reconstructing externally perceived visual content from neural signals.
By Zhenhang Li, Xin Zhou, Hao Deng, Lijun Yin
arXiv:2609.12668v1 Announce Type: new
Abstract: Recent deepfake detection studies increasingly suggest remote photoplethysmography (rPPG) signals as an authenticity cue. However, existing benchmarks...
By Chenxi Yang, Yassine Ouzar, Larbi Boubchir
arXiv:2608. 08873v1 Announce Type: cross Abstract: Facial Emotion Recognition (FER) is an important task that has significant implications across various fields such as biometrics, health, and human-computer interaction.
By Aya Manel Zitouni, Aicha Zenakhri, Karim Haroun, Larbi Boubchir
arXiv:2607. 12774v1 Announce Type: cross Abstract: This article presents our results for the 11th Affective Behavior Analysis in-the-Wild (ABAW) competition.
By Aleksei Bakin, Andrey V. Savchenko
HUG‑VIS is a unified multimodal benchmark for human‑centered visual intelligence, comprising 8,400 half‑body videos of 30 professional actors performing 280 emotion‑action prompts in Mandarin. The dataset provides synchronized video, audio, text, and alpha mattes for four tasks—human emotion recognition, video generation, voice cloning, and video matting—allowing evaluation of both open‑ and closed‑source models under a zero‑shot protocol. Results reveal that linguistic cues dominate emotion recognition, visual affect is weakest, and that automatic metrics and human judgments diverge in generation and cloning tasks, while motion‑related boundary fidelity remains a key challenge for matting.
By Fei Ma, Zebang Cheng, Minghui Li, Hongbo Xu, Yuyong Tan, Yihua Shao, Hanling Wang, Zhou Liu, Yuqing Gao, Dong Wang, Long Ma, Laizhong Cui, Nicu Sebe, Qi Tian
The paper introduces Affect-Prototype Guided Fusion (APCF), a framework for open‑vocabulary multimodal emotion recognition that handles incomplete and unsynchronized modal data. APCF builds an affect‑prototype library to model how different emotions contribute across modalities, enabling dynamic fusion of available features. The fused representations are then decoded by an LLM to generate open‑vocabulary emotion labels, achieving superior performance on OV‑MERD+ and MER‑FG datasets compared to existing methods.
By Yichi Zhang, Shenyue Wang, Jing Luo, Chunyang Yu, Xinyu Yang