The paper introduces an incongruence score that quantifies disagreement between a speaker’s verbal content and vocal expression in a shared valence‑arousal‑dominance space, validated against human judgments. Using this metric, the authors evaluate three audio‑capable large language models across acted, conversational, and spontaneous speech datasets, finding that higher incongruence consistently increases error rates when audio is present. The study also shows that supervised classifiers behave oppositely, suggesting the challenge lies in how models fuse multimodal evidence rather than in the signal itself, and that the score remains predictive even without an emotion lexicon.
By Fengji Jing, Guy Laban
The paper introduces CLEAR, a framework for Multimodal Emotion Recognition in Conversation that couples contextual representations with local discriminative cues. It replaces emotion label generation with continuous internal representations from a vision‑language model and trains a classifier on these representations. Sample‑adaptive residual coupling is used to integrate textual and acoustic evidence from the target utterance into the contextual representation, improving emotion recognition performance on IEMOCAP and MELD datasets.
By Linan Zhu, Zihao Zhai, Mengfei Xu, Xiao Han, Yuqian Fu, Xiangfan Chen, Zihang Zhang
TalkFa introduces a unified benchmark for Farsi dialogue generation and understanding, comprising three datasets: WIKI‑FADIAL (4.2K Wikipedia‑grounded dialogues), DAILYDIALOG‑FA (6.6K dialogues with dialogue‑act and emotion annotations), and PLAYDIAL‑FA (2.1K theatrical dialogues with sentiment labels). All dialogues are curated through multi‑stage review by native speakers, ensuring high quality. Experiments show that LoRA fine‑tuning improves generation performance with less data, while specific models excel on classification tasks, and human evaluation confirms the benchmark’s reliability.
By Neda Jamshidi, Kamyar Zeinalipour, Fahimeh Akbari, Monica Bianchini, Marco Maggini, Marco Gori
The paper compares text‑based and feature‑based models for recognizing compound emotions in real‑world videos. It proposes textualizing non‑verbal cues from audio and visual modalities into text to leverage large language models, while feature‑based models directly combine extracted multimodal features. Experiments on the C‑EXPR‑DB dataset show that feature‑based models outperform textualization in the wild, though textual models can excel when rich transcripts are available.
By Nicolas Richet, Soufiane Belharbi, Haseeb Aslam, Meike Emilie Schadt, Manuela Gonz\'alez-Gonz\'alez, Gustave Cortal, Alessandro Lameiras Koerich, Marco Pedersoli, Alain Finkel, Simon Bacon, Eric Granger
SpeechEQ is a new benchmark that evaluates the sociolinguistic reasoning of Speech‑Language Models (SLMs) by using a dataset of 2,265 dialogues covering 15 Emotional Quotient (EQ) subscales from EQ‑i 2.0 theory. The framework introduces a multi‑turn evaluation protocol and a Spoken EQ (SEQ) score inspired by human EQ assessments. Experiments reveal that current multimodal models suffer from a text‑reliant shortcut, an alignment‑induced safety trap, and contextual amnesia, limiting their emotional awareness in spoken interaction.
By Liang-Yuan Wu, Zih-Ching Chen, Tongshuang Wu, Chao-Han Huck Yang, Hua Shen
arXiv:2609.05806v1 Announce Type: new
Abstract: Emotion Recognition in Conversations (ERC) aims to identify speakers' emotions in multi-turn dialogue. Accurate emotion recognition can support a wide...
By Amir Ben Khalifa, Fanny Bezancon, Amine Trabelsi, Bessam Abdulrazak