The paper introduces Evidential Multimodal Transformers with Dempster-Shafer Fusion (EMT-DS) for emotion recognition, which models predictions as Dirichlet distributions to separate epistemic and aleatoric uncertainty. EMT-DS fuses modality-specific belief masses using Dempster's Rule, explicitly quantifying conflict and producing more diffuse predictions when evidence is contradictory. Across four benchmarks, EMT-DS outperforms baselines, reduces calibration error, remains robust to noise, and its conflict metric can flag misclassifications for selective prediction.
arXiv:2607. 08573v1 Announce Type: new Abstract: Multimodal emotion and sentiment recognition is commonly addressed by early fusion, which concatenates modalities before classification, or late fusion, which combines independently trained unimodal predictors.
The paper introduces a multimodal emotion recognition system that integrates facial expressions, speech, text, and body movement to aid clinicians in psychological evaluation. While individual modalities perform well on benchmarks, their real‑world accuracy drops to 82.97%–87.64%, whereas the fused multimodal system reaches 96.43% accuracy with no state below 92.31%. These findings demonstrate that multimodal fusion enhances robustness in realistic settings, supporting more reliable evaluator assistance.
Conventional face recognition relies on static appearance cues and degrades in unconstrained settings with expression variation, occlusion, and poor lighting. We hypothesize that audiovisual expression dynamics carry identity-discriminative information complementary to static appearance, and that extracting this signal requires multimodal representations robust to the variable input quality of in-the-wild video.
arXiv:2608. 07867v1 Announce Type: new Abstract: Multimodal emotion recognition often treats self-reported labels as reliable supervision while overlooking self-report unreliability and cross-modal conflict.
arXiv:2606. 02679v1 Announce Type: new Abstract: Multimodal systems often benefit from combining information across language, sound, and visual streams, but this benefit is not guaranteed.
The paper introduces the Modality Discrepancy Transformer (MDT), a model designed to detect ambivalence and hesitancy in clinical videos by capturing cross‑modal disagreement across facial, vocal, and linguistic signals. MDT expands a 6‑token representation to 9 tokens that include modality embeddings, absolute‑difference features, and Hadamard‑product discrepancy features, which are processed through Transformer self‑attention with FiLM‑based text conditioning and LoRA fine‑tuning. On the BAH dataset from the 3rd ABAW Challenge, MDT achieves a Macro F1 score of 0.7408 on the labelled test split and 0.7368 on the private leaderboard, surpassing the strongest baseline by over 10 points while training in under 20 minutes on a single GPU.
The paper introduces EmoSpeechBrain, a multimodal emotion recognition framework that fuses EEG and speech signals. It employs differential attention in the EEG encoder to cancel shared noise and an attention-based gating adapter to align modalities and weight their contributions. Experiments on PME4 and EAV datasets show up to 12.9% accuracy improvement over other EEG encoders and surpass unimodal baselines by up to 23.1%.
arXiv:2608. 15999v1 Announce Type: new Abstract: Automatic emotion assessment can benefit from combining neural and behavioral signals, but many multimodal approaches rely on separate, modality-specific feature-extraction pipelines before fusion.
arXiv:2607. 20742v1 Announce Type: new Abstract: Multimodal learning is a robust approach to improve predictive performance in applications such as medical prognosis.
The paper introduces the Evidential Missing Modality Survival Fusion (EMMS) model, which predicts survival outcomes using multimodal data even when some modalities are missing. EMMS applies Dempster‑Shafer theory and Gaussian Random Fuzzy Numbers to fuse information, accounting for both aleatoric and epistemic uncertainty and the reliability of each modality. Experiments on four cancer datasets show that EMMS achieves state‑of‑the‑art performance while providing calibrated, interpretable uncertainty estimates without extra computational cost.
arXiv:2607. 18336v1 Announce Type: cross Abstract: Multimodal emotion recognition in conversation (MERC) can leverage multimodal and contextual cues to boost recognition performance.
The paper introduces Linear Discriminant Tree Ensembles (LDT, LDF, LDAB) for multimodal affect and behaviour classification, combining text, audio, and visual streams. It uses tokenization, concept clustering, and tree-based routing to balance accuracy and interpretability, and proposes a modified feature importance metric that reduces negative class bias. The ensembles outperform Multimodal Transformers and Interpretable Multimodal Routing in F1-mod and accuracy, and their feature importance aligns better with human annotations on IEMOCAP and CMU-MOSI datasets.