← Back to all news
arXiv AI By Ngo Truong Dinh, Tung-Lam Bui, Chi-Trung Duong, Vien Nguyen Thi, Viet-Anh Nguyen, Phuc-Lu Le

RAFM-SER++: A Lightweight Multimodal Emotion Recognition Framework for Real-Time Behavioral Monitoring in Surveillance Systems

Read the original on arXiv AI →

The paper introduces RAFM-SER++, a lightweight multimodal speech emotion recognition framework designed for real‑time surveillance systems. It replaces heavy bidirectional cross‑modal transformers with an asymmetric Residual Attention Fusion Mechanism that injects affective speech cues into text representations via a one‑directional residual attention pathway. Experiments on IEMOCAP and ESD show RAFM‑SER++ outperforms the HuBERT‑Base baseline and MemoCMT, reducing trainable parameters by over 60%, achieving 79.60 it/s inference speed, and reaching BACC scores of 81.10% on IEMOCAP and 95.39% on ESD.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Sep 10

MVFA: A Multi-View Text-Guided Multimodal Fusion LLM Adapter for Sentiment Analysis and Emotion Recognition

arXiv:2609.06188v1 Announce Type: new Abstract: Multimodal sentiment analysis and emotion recognition in conversations demand effective modeling of heterogeneous interactions across textual, acoustic...

By Pengfei Shao, Jisheng Dang, Jiawen Fang, Ning Liu, Wencan Zhang, Bimei Wang, Jingwen Zhao, Jianhuang Lai, Qi Tian, Tat-Seng Chua
llmsnlpfine-tuningmultimodalbenchmarks
More like this →
arXiv AI
Sep 7

Enhancing Multimodal Emotion Recognition via Multi-Feature Encoding and Attention-Based Fusion

The paper introduces a multimodal emotion recognition framework that combines audio and visual feature extraction with an attention-based fusion strategy. Audio features include Wav2Vec2 embeddings, MFCCs, and statistical acoustic descriptors, fused via a BiLSTM, while video features are extracted using a ResNet50-BiLSTM architecture. A multi-head attention mechanism fuses these modalities, and experiments on MELD and IEMOCAP show significant accuracy and robustness gains, especially in unbalanced data settings.

By Xu Lin, Ke Wang, Hui Kang, Xinying Wang
ragmultimodalbenchmarks
More like this →
arXiv AI
Jun 2

UF-AMA: A unified framework for cross-domain emotion recognition via adaptive multimodal alignment

arXiv:2606. 00170v1 Announce Type: cross Abstract: In recent years, emotion recognition based on physiological signals such as electroencephalogram (EEG) has gained considerable attention, as internal physiological data offer greater objectivity and reliability compared to external behavioral data like facial expressions.

By Zheng Wang, Shuo Wang, Junhong Wang
llmsfine-tuningefficiencymultimodalbenchmarkssafety
More like this →
arXiv Computation and Language
Aug 31

OmniFusion: Simultaneous Multilingual Multimodal Translations via Modular Fusion

OmniFusion is an end‑to‑end multilingual multimodal translation system that fuses a pretrained multimodal foundation model (Omni 2.5‑7B) with a translation large language model (SeedX PPO‑7B). By connecting hidden states from multiple layers of the multimodal model to the translation LLM, OmniFusion can translate speech, speech‑and‑image, and text‑and‑image inputs while reducing simultaneous speech‑translation latency by about one second compared to cascaded pipelines. The approach improves overall translation quality by leveraging both audio and visual context.

By Sai Koneru, Matthias Huck, Jan Niehues
llmsreinforcement-learningmultimodal
More like this →