The paper introduces Affect-Prototype Guided Fusion (APCF), a framework for open‑vocabulary multimodal emotion recognition that handles incomplete and unsynchronized modal data. APCF builds an affect‑prototype library to model how different emotions contribute across modalities, enabling dynamic fusion of available features. The fused representations are then decoded by an LLM to generate open‑vocabulary emotion labels, achieving superior performance on OV‑MERD+ and MER‑FG datasets compared to existing methods.
arXiv:2601. 07565v2 Announce Type: replace-cross Abstract: Multimodal emotion understanding requires the integration of heterogeneous data sources, including text, audio, and visual modalities, while simultaneously addressing discrete emotion recognition and continuous sentiment analysis.
arXiv:2609.06188v1 Announce Type: new Abstract: Multimodal sentiment analysis and emotion recognition in conversations demand effective modeling of heterogeneous interactions across textual, acoustic...
Cross-Modal Emotion Understanding: A Transformer-GAT Approach for Dialogue Emotion Recognition proposes a hybrid framework that combines a Transformer and a Graph Attention Network to capture both global semantic information and fine-grained relationships between modalities. The model is evaluated on the IEMOCAP and MELD datasets, achieving weighted F1 scores of 72.45% and 77.37%, respectively, and surpasses state‑of‑the‑art methods. These results suggest that integrating multimodal features with balanced global and local context modeling can provide deeper emotional insights for dialogue emotion recognition.
In multimodal emotion recognition (MER), human affective states are inferred by integrating complementary cues from multiple modalities. In audio-text MER, affective cues are often entangled with spea...
The paper introduces a multimodal emotion recognition framework that combines audio and visual feature extraction with an attention-based fusion strategy. Audio features include Wav2Vec2 embeddings, MFCCs, and statistical acoustic descriptors, fused via a BiLSTM, while video features are extracted using a ResNet50-BiLSTM architecture. A multi-head attention mechanism fuses these modalities, and experiments on MELD and IEMOCAP show significant accuracy and robustness gains, especially in unbalanced data settings.
arXiv:2610. 29430v1 Announce Type: new Abstract: Multimodal multi-label emotion recognition is important for affect-aware systems, but models trained with complete text, visual, and acoustic streams often degrade sharply when some streams are unavailable at deployment.
arXiv:2608. 20019v1 Announce Type: new Abstract: Incomplete multimodal sentiment analysis has garnered significant attention in recent years.
arXiv:2606. 02679v1 Announce Type: new Abstract: Multimodal systems often benefit from combining information across language, sound, and visual streams, but this benefit is not guaranteed.
BiCFlow-MER introduces a conditional-flow framework for audio-text multimodal emotion recognition, treating the task as generative evidence transport within a structured emotion space. It disentangles emotion-oriented evidence from speaker style and lexical content, creating a conflict-aware affective condition that guides bidirectional rectified flow to an explicit emotion-space endpoint. The model verifies candidate emotions via adaptive prototype-cloud scoring and backward class-to-condition consistency, achieving superior performance on IEMOCAP, MELD, and the zero-shot CASE benchmark.
arXiv:2608. 04013v1 Announce Type: cross Abstract: Recent advances in Multimodal Emotion Recognition in Conversations (MERC) highlight its reliance on complete multimodal inputs.
The paper introduces CLEAR, a framework for Multimodal Emotion Recognition in Conversation that couples contextual representations with local discriminative cues. It replaces emotion label generation with continuous internal representations from a vision‑language model and trains a classifier on these representations. Sample‑adaptive residual coupling is used to integrate textual and acoustic evidence from the target utterance into the contextual representation, improving emotion recognition performance on IEMOCAP and MELD datasets.
Understanding both expressed and evoked emotions is critical for multimodal large language models (MLLMs) to achieve comprehensive affect-aware interactions. However, existing benchmarks typically examine expressed and evoked emotions in isolation or are constrained to coarse-grained and incomplete affective characterizations.