Conventional face recognition relies on static appearance cues and degrades in unconstrained settings with expression variation, occlusion, and poor lighting. We hypothesize that audiovisual expression dynamics carry identity-discriminative information complementary to static appearance, and that extracting this signal requires multimodal representations robust to the variable input quality of in-the-wild video.
arXiv:2607. 05019v1 Announce Type: new Abstract: In multimodal classification, late-fusion approaches classify concatenated modality-specific features extracted by unimodal neural networks.
By Ilya Burenko, Dmitry Vetrov
arXiv:2606. 02679v1 Announce Type: new Abstract: Multimodal systems often benefit from combining information across language, sound, and visual streams, but this benefit is not guaranteed.
By Jiyuan Liu, Liangwei Nathan Zheng, Wei Emma Zhang, Xinpei Wang, Weitong Chen
arXiv:2606. 26473v1 Announce Type: new Abstract: Many multimodal systems estimate the reliability of each modality and weight their contributions to the final prediction.
By Jaden Moon, Arvind Pillai, Andrew Campbell
arXiv:2609.06188v1 Announce Type: new
Abstract: Multimodal sentiment analysis and emotion recognition in conversations demand effective modeling of heterogeneous interactions across textual, acoustic...
By Pengfei Shao, Jisheng Dang, Jiawen Fang, Ning Liu, Wencan Zhang, Bimei Wang, Jingwen Zhao, Jianhuang Lai, Qi Tian, Tat-Seng Chua
The paper introduces the Modality Discrepancy Transformer (MDT), a model designed to detect ambivalence and hesitancy in clinical videos by capturing cross‑modal disagreement across facial, vocal, and linguistic signals. MDT expands a 6‑token representation to 9 tokens that include modality embeddings, absolute‑difference features, and Hadamard‑product discrepancy features, which are processed through Transformer self‑attention with FiLM‑based text conditioning and LoRA fine‑tuning. On the BAH dataset from the 3rd ABAW Challenge, MDT achieves a Macro F1 score of 0.7408 on the labelled test split and 0.7368 on the private leaderboard, surpassing the strongest baseline by over 10 points while training in under 20 minutes on a single GPU.
By Shiyu Luo, Yu Wang, Jiawen Huang, Zhaoxiang Xiao, Chenxi Huang, Qi Zhang, Bin Liu
arXiv:2608. 03611v1 Announce Type: new Abstract: Multimodal Sentiment Analysis (MSA) integrates text, audio, and vision to infer human affect, yet real-world multimodal observations are often incomplete.
By Chunlei Meng, Jacqueline J. Pang, Pengbin Feng, Zhenyu Yu, Chun Ouyang, Zhongxue Gan
The paper introduces Linear Discriminant Tree Ensembles (LDT, LDF, LDAB) for multimodal affect and behaviour classification, combining text, audio, and visual streams. It uses tokenization, concept clustering, and tree-based routing to balance accuracy and interpretability, and proposes a modified feature importance metric that reduces negative class bias. The ensembles outperform Multimodal Transformers and Interpretable Multimodal Routing in F1-mod and accuracy, and their feature importance aligns better with human annotations on IEMOCAP and CMU-MOSI datasets.
By Mojtaba Moattari
Traits Run Deeper introduces a personality assessment framework that tailors multimodal fusion to each trait dimension. It comprises a Multimodal Foundation Representation module that uses psychology-informed semantic templates, a Trait-Specific Modality Fusion module that asymmetrically fuses modalities to reduce cross‑modal interference, and a Distribution‑Calibrated Personality Regression module that corrects label imbalance. The approach achieves a ~25% reduction in mean squared error on the AVI Challenge 2026 validation set and wins the Personality Assessment Track.
By Jia Li, Qian Chen, Wei Wang, Xinyu Li, Zhenzhen Hu, Dongsheng Shao, Richang Hong, Meng Wang
arXiv:2607. 27289v1 Announce Type: new Abstract: The promise of multimodal fusion lies in combining complementary sources of evidence, yet more evidence does not always yield a better prediction.
By Yu Chang, Anzhe Cheng, Chenwei Wu, Zhuoran Wang, Jiahao Chen, Tamoghna Chattopadhyay, Sophia I. Thomopoulos, Paul M. Thompson, Liyue Shen, Paul Bogdan
arXiv:2601. 07565v2 Announce Type: replace-cross Abstract: Multimodal emotion understanding requires the integration of heterogeneous data sources, including text, audio, and visual modalities, while simultaneously addressing discrete emotion recognition and continuous sentiment analysis.
By Jiaqi Qiao, Xinran Li, Yifan Lyu, Xiujuan Xu, Liu Yu
arXiv:2606.11269v2 Announce Type: replace
Abstract: Personality assessment aims to infer stable traits from dynamic behaviors across modalities like language, voice, and facial expressions. Existing...
By Jia Li, Qian Chen, Wei Wang, Xinyu Li, Zhenzhen Hu, Dongsheng Shao, Richang Hong, Meng Wang