ReH-FUSE is a reliability‑aware hierarchical fusion framework for multimodal emotion recognition in conversation. It uses a decision‑level router to first compare the relative preference between text and audio, then balances this unimodal mixture with a cross‑modal expert, thereby separating unimodal competition from cross‑modal selection. Experiments on IEMOCAP and MELD show that ReH-FUSE achieves state‑of‑the‑art weighted and macro F1 scores, and ablation studies confirm that learned routing outperforms uniform expert averaging and benefits from cross‑modal interaction.
arXiv:2607. 18336v1 Announce Type: cross Abstract: Multimodal emotion recognition in conversation (MERC) can leverage multimodal and contextual cues to boost recognition performance.
The paper introduces TAME, a Text‑Anchored Modality Experts framework that treats language as a fixed, always‑active expert and routes audio‑textual and video‑textual cross‑modal branches as dynamic auxiliary experts. It incorporates lightweight Prophet Heads to estimate sample‑level uncertainty for each modality, guiding a reliability‑aware routing mechanism for expert aggregation. Experiments on CMU‑MOSI, CMU‑MOSEI, and CH‑SIMS show that TAME achieves state‑of‑the‑art performance while reducing parameters by up to 78.7% and FLOPs by up to 41.2% compared to a mixture‑of‑experts baseline.
arXiv:2601. 07565v2 Announce Type: replace-cross Abstract: Multimodal emotion understanding requires the integration of heterogeneous data sources, including text, audio, and visual modalities, while simultaneously addressing discrete emotion recognition and continuous sentiment analysis.
The paper introduces RAFM-SER++, a lightweight multimodal speech emotion recognition framework designed for real‑time surveillance systems. It replaces heavy bidirectional cross‑modal transformers with an asymmetric Residual Attention Fusion Mechanism that injects affective speech cues into text representations via a one‑directional residual attention pathway. Experiments on IEMOCAP and ESD show RAFM‑SER++ outperforms the HuBERT‑Base baseline and MemoCMT, reducing trainable parameters by over 60%, achieving 79.60 it/s inference speed, and reaching BACC scores of 81.10% on IEMOCAP and 95.39% on ESD.
arXiv:2608.30726v1 Announce Type: new Abstract: Multimodal Sentiment Analysis (MSA) is a fundamental component of affective computing that aims to decipher complex emotional states by integrating ver...
arXiv:2608. 04013v1 Announce Type: cross Abstract: Recent advances in Multimodal Emotion Recognition in Conversations (MERC) highlight its reliance on complete multimodal inputs.
Reliability-aware Cross-sample Enhancement (RCE) is a framework for multimodal sentiment analysis that tackles noise and missing modalities by first applying an adaptive variational information bottleneck to compress unreliable modality information. It then retrieves high‑confidence, semantically consistent neighbors from a large candidate pool to enrich current representations, and finally fuses cross‑modal interactions through a multilevel reliability‑aware mechanism. Experiments show RCE consistently outperforms state‑of‑the‑art methods in full, noisy, and missing‑modality scenarios.
The paper introduces CLEAR, a framework for Multimodal Emotion Recognition in Conversation that couples contextual representations with local discriminative cues. It replaces emotion label generation with continuous internal representations from a vision‑language model and trains a classifier on these representations. Sample‑adaptive residual coupling is used to integrate textual and acoustic evidence from the target utterance into the contextual representation, improving emotion recognition performance on IEMOCAP and MELD datasets.
arXiv:2609.09924v1 Announce Type: new Abstract: Emotion Recognition in Conversations (ERC) requires integrating heterogeneous textual, audio, and visual cues while accounting for conversational conte...
arXiv:2606. 02679v1 Announce Type: new Abstract: Multimodal systems often benefit from combining information across language, sound, and visual streams, but this benefit is not guaranteed.
The article surveys multi‑turn conversational AI, highlighting its shift from isolated text prompts to sustained, multimodal interactions that involve clarifying goals, revising requests, and switching topics. It reviews literature across text‑only dialogue, AudioLLMs, multimodal and omni‑modal systems, and tool‑augmented agents, organizing findings around datasets, models, training, evaluation, and cross‑cutting challenges. The analysis reveals that while multimodal perception and action have progressed rapidly, systems still struggle with persistent memory, cross‑turn grounding, full‑duplex interaction, robust evaluation, and cultural alignment.
Cross-Modal Emotion Understanding: A Transformer-GAT Approach for Dialogue Emotion Recognition proposes a hybrid framework that combines a Transformer and a Graph Attention Network to capture both global semantic information and fine-grained relationships between modalities. The model is evaluated on the IEMOCAP and MELD datasets, achieving weighted F1 scores of 72.45% and 77.37%, respectively, and surpasses state‑of‑the‑art methods. These results suggest that integrating multimodal features with balanced global and local context modeling can provide deeper emotional insights for dialogue emotion recognition.