← Back to all news
arXiv Computation and Language By TingTing Gu, JinCheng Luo, WenQiu Guo

Multimodal Ambivalence and Hesitancy Recognition with Bidirectional Audio--Text Cross-Attention

Read the original on arXiv Computation and Language →

The paper introduces a multimodal framework that jointly processes full‑video transcripts and speech waveforms to recognize ambivalence and hesitancy (A/H) at the video level. It uses pre‑trained multilingual BERT for text and wav2vec 2.0 for audio, projecting both into a shared space and applying two stacked bidirectional cross‑attention blocks to fuse linguistic and acoustic cues without explicit temporal alignment. The resulting representations are aggregated and combined via concatenation, element‑wise products, and absolute differences, achieving 72.19 % accuracy and 77.95 % A/H F1‑score on the Behavioural Ambivalence/Hesitancy dataset.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv Computer Vision
Sep 2

Audio-Text Cross-Attention with Psycholinguistic Support Features for Ambivalence/Hesitancy Recognition

arXiv:2607.13345v2 Announce Type: replace Abstract: We present a frame-independent audio-text system for the 3rd Ambivalence/Hesitancy Video Recognition Challenge at the 11th Affective & Behavior Ana...

By Luiz F. B. F. Martins, Rodrigo W. Pisaia, Matheus M. Girardi, Isabella V. Berkembrock, Jo\~ao A. Almeida, Andre G. Hochuli, Rayson Laroca, Alceu S. Britto Jr
ragbenchmarks
More like this →
arXiv Computation and Language
Sep 18

Modality Discrepancy Transformer for Ambivalence and Hesitancy Recognition

The paper introduces the Modality Discrepancy Transformer (MDT), a model designed to detect ambivalence and hesitancy in clinical videos by capturing cross‑modal disagreement across facial, vocal, and linguistic signals. MDT expands a 6‑token representation to 9 tokens that include modality embeddings, absolute‑difference features, and Hadamard‑product discrepancy features, which are processed through Transformer self‑attention with FiLM‑based text conditioning and LoRA fine‑tuning. On the BAH dataset from the 3rd ABAW Challenge, MDT achieves a Macro F1 score of 0.7408 on the labelled test split and 0.7368 on the private leaderboard, surpassing the strongest baseline by over 10 points while training in under 20 minutes on a single GPU.

By Shiyu Luo, Yu Wang, Jiawen Huang, Zhaoxiang Xiao, Chenxi Huang, Qi Zhang, Bin Liu
llmsragfine-tuningmultimodalbenchmarks
More like this →
arXiv AI
Jul 17

Team RAS in 11th ABAW Competition: Multimodal Ambivalence Recognition Approach

arXiv:2607. 14702v1 Announce Type: cross Abstract: Automatic recognition of ambivalence and hesitancy is challenging because these states may be expressed through inconsistent linguistic, acoustic, facial, and contextual patterns, while top-performing systems often rely on computationally expensive ensembles.

By Elena Ryumina (St. Petersburg Federal Research Center of the Russian Academy of Sciences), Maxim Markitantov (St. Petersburg Federal Research Center of the Russian Academy of Sciences), Alexandr Axyonov (St. Petersburg Federal Research Center of the Russian Academy of Sciences), Fedor Shchetinin (HSE University, St. Petersburg, Russia), Timur Abdulkadirov (St. Petersburg Federal Research Center of the Russian Academy of Sciences), Dmitry Ryumin (St. Petersburg Federal Research Center of the Russian Academy of Sciences), Alexey Karpov (St. Petersburg Federal Research Center of the Russian Academy of Sciences)
multimodal
More like this →