arXiv Computer Vision
A Comprehensive Review of Multimodal Facial State Analysis: Tasks, Methods, and Resources
Read the original on arXiv Computer Vision →The Flow has not summarised this story yet — read it at arXiv Computer Vision.
Related stories
arXiv AI
Jun 30
LLM-based Multimodal Personality Recognition via Facial Action Unit-Text Semantic Fusion
arXiv:2606. 29900v1 Announce Type: cross Abstract: Personality recognition in asynchronous video interviews (AVIs) has become increasingly important due to their widespread adoption in modern recruitment.
Hugging Face Trending Papers
Jun 29
LLM-based Multimodal Personality Recognition via Facial Action Unit-Text Semantic Fusion
Personality recognition in asynchronous video interviews (AVIs) has become increasingly important due to their widespread adoption in modern recruitment. Existing approaches often rely on large language models (LLMs) to analyze textual responses of interviewees in AVI.
arXiv Computer Vision
Aug 28
HUG-VIS: A Multimodal Benchmark for Human-centered Understanding and Generation in Visual Intelligence
HUG‑VIS is a unified multimodal benchmark for human‑centered visual intelligence, comprising 8,400 half‑body videos of 30 professional actors performing 280 emotion‑action prompts in Mandarin. The dataset provides synchronized video, audio, text, and alpha mattes for four tasks—human emotion recognition, video generation, voice cloning, and video matting—allowing evaluation of both open‑ and closed‑source models under a zero‑shot protocol. Results reveal that linguistic cues dominate emotion recognition, visual affect is weakest, and that automatic metrics and human judgments diverge in generation and cloning tasks, while motion‑related boundary fidelity remains a key challenge for matting.
arXiv AI
Sep 7
Enhancing Multimodal Emotion Recognition via Multi-Feature Encoding and Attention-Based Fusion
The paper introduces a multimodal emotion recognition framework that combines audio and visual feature extraction with an attention-based fusion strategy. Audio features include Wav2Vec2 embeddings, MFCCs, and statistical acoustic descriptors, fused via a BiLSTM, while video features are extracted using a ResNet50-BiLSTM architecture. A multi-head attention mechanism fuses these modalities, and experiments on MELD and IEMOCAP show significant accuracy and robustness gains, especially in unbalanced data settings.
arXiv Computer Vision
Sep 4
Occlusion-Robust Multimodal Emotion Recognition in VR via Fusion of Facial Images and EMG
The paper presents a method for emotion recognition in virtual reality where head‑mounted displays occlude the upper face. By fusing lower‑face video with electromyography (EMG) signals from the occluded upper face, the authors achieve a 51% macro‑F1 score across seven emotional categories, outperforming image‑only and EMG‑only baselines. A new synchronized multimodal dataset from 20 participants is introduced and will be shared under an ethical‑use agreement.
arXiv AI
Jul 17
InCarEmo: A Multimodal Dataset for In-Cabin Emotion Recognition and Driver State Monitoring
arXiv:2607. 14683v1 Announce Type: new Abstract: Understanding driver emotion and state is critical for the next generation of intelligent in-cabin systems that ensure safety and enhance human-vehicle interaction.