arXiv AI June 2, 2026 By Zhou Yang, Yueyi Yang

Beyond the Mouth: Upper-Face Affective Cues in Audiovisual Sentence Recognition under Acoustic Uncertainty

arXiv:2606. 00670v1 Announce Type: cross Abstract: Face-to-face speech comprehension is inherently multimodal, integrating acoustic signals with visible articulation, facial expression, head motion, and other socially relevant cues.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

multimodal

arXiv AI

3d ago

From Speech to Editable Concepts: Probing Emotion Recognition with Concept Bottleneck Models

arXiv:2609.39453v1 Announce Type: cross Abstract: Speech emotion recognition (SER) is the task of assigning emotion labels to utterances. Early systems relied on acoustic features, whereas recent app...

By Hezhao Zhang, Thomas Hain

llms computer-vision fine-tuning safety

arXiv Machine Learning

Sep 16

EMODY Flow: Emotion-Aware Audio-Driven Full-Body Motion Generation

EMODY Flow is a lightweight flow‑matching framework that generates synchronized full‑body motion and facial expressions conditioned on speech and emotion. It attaches to a frozen Qwen‑3 Omni model, reusing its audio codecs to drive two parallel DiT generators for SMPL‑X body pose and FLAME facial expressions. An auxiliary emotion classifier at training time restores emotion sensitivity, enabling EMODY Flow to achieve state‑of‑the‑art gesture quality on BEAT2 and zero‑shot facial animation on TFHP, with significant improvements in FGD, Beat Correlation, and Diversity metrics.

By Harsh Kumar Agarwal, Xavier Alameda-Pineda, Olivier Perrotin

llms rag agents robotics fine-tuning multimodal

arXiv Computation and Language

Sep 1

VocalAffectBench: Evaluating Vocal Emotion Recognition in AI Audio Models

arXiv:2608.28932v1 Announce Type: new Abstract: Voice products increasingly need affective cues that are present in speech but absent from transcripts. We introduce VocalAffectBench, a public, test-o...

By Models Luc Debaupte, Tyler Baumgartner, Brandon Tai, Candice Fan, Bill Wang, Yi Zhong

agents benchmarks

arXiv Computer Vision

Aug 24

EmotionDialogCN: A Spontaneous Multimodal Dataset for Mandarin Emotional Dialogue

arXiv:2608.20905v1 Announce Type: new Abstract: Face-to-face audiovisual interaction is central to human communication, conveying rich emotional and social cues. However, existing multimodal dialogue...

By Yi Zheng, Yifan Xu, Yan Zhou, Hejia Chen, Chunyu Qiang, Xiaoqiang Liu, Xiaohan Li, Shenze Huang, Yue Zhang, Guoying Zhao, Pengfei Wan

multimodal safety

arXiv Machine Learning

Jun 2

Sympatheia: Emotionally Adaptive Voice Assistant with Continuous Affect Conditioning

arXiv:2606. 00851v1 Announce Type: cross Abstract: Empathetic spoken dialogue systems must infer a user's emotional state to respond appropriately, yet everyday speech often carries weak, neutral, or ambiguous affective cues.

By Sukru Samet Dindar, Riki Shimizu, Xilin Jiang, Nima Mesgarani

multimodal safety

arXiv Computation and Language

Sep 23

Enriching Speech Emotion Representations with Conversational Context

The paper introduces ACERT, a module that incorporates a flexible-length window of conversational context to enhance Speech Emotion Recognition (SER). By capturing emotional evolution across utterances, ACERT outperforms state‑of‑the‑art methods on IEMOCAP, sets a new context‑aware benchmark on SAFE, and achieves strong results on MELD. Ablation studies attribute ACERT’s improvements to emotional and conversational continuity rather than speaker identity or acoustic conditions.

By Arthur Peuvot, Romaric Besan\c{c}on, Ga\"el de Chalendar, Bianca Vieru, Ioana Vasilescu

benchmarks