The paper introduces a multimodal framework that jointly processes full‑video transcripts and speech waveforms to recognize ambivalence and hesitancy (A/H) at the video level. It uses pre‑trained multilingual BERT for text and wav2vec 2.0 for audio, projecting both into a shared space and applying two stacked bidirectional cross‑attention blocks to fuse linguistic and acoustic cues without explicit temporal alignment. The resulting representations are aggregated and combined via concatenation, element‑wise products, and absolute differences, achieving 72.19 % accuracy and 77.95 % A/H F1‑score on the Behavioural Ambivalence/Hesitancy dataset.
arXiv:2607.13345v2 Announce Type: replace Abstract: We present a frame-independent audio-text system for the 3rd Ambivalence/Hesitancy Video Recognition Challenge at the 11th Affective & Behavior Ana...
We present an audio-text system for the Ambivalence/Hesitancy Video Recognition Challenge of the 11th ABAW Competition. The method excludes visual frames and represents each video as overlapping 5-second windows aligned with transcript timestamps.
The paper introduces the Modality Discrepancy Transformer (MDT), a model designed to detect ambivalence and hesitancy in clinical videos by capturing cross‑modal disagreement across facial, vocal, and linguistic signals. MDT expands a 6‑token representation to 9 tokens that include modality embeddings, absolute‑difference features, and Hadamard‑product discrepancy features, which are processed through Transformer self‑attention with FiLM‑based text conditioning and LoRA fine‑tuning. On the BAH dataset from the 3rd ABAW Challenge, MDT achieves a Macro F1 score of 0.7408 on the labelled test split and 0.7368 on the private leaderboard, surpassing the strongest baseline by over 10 points while training in under 20 minutes on a single GPU.
arXiv:2512.07571v3 Announce Type: replace Abstract: This paper presents a simple method that allows to easily enhance textual pre-trained large language models with speech information, when fine-tune...
arXiv:2607. 16635v1 Announce Type: cross Abstract: We present TellTale, a text-only approach to ambivalence/hesitancy (A/H) recognition in interview videos, evaluated on the BAH dataset as part of the 3rd A/H Video Recognition Challenge (11th ABAW Workshop, ECCV 2026).
arXiv:2607. 14702v1 Announce Type: cross Abstract: Automatic recognition of ambivalence and hesitancy is challenging because these states may be expressed through inconsistent linguistic, acoustic, facial, and contextual patterns, while top-performing systems often rely on computationally expensive ensembles.
Omnilingual-MsASR is a new benchmark designed to evaluate omni‑modal large language models on long‑form, multi‑speaker automatic speech recognition across six languages, with recordings up to 30 minutes and up to 20 speakers. The benchmark provides parallel audio and video inputs to test cross‑modal consistency and the models’ ability to use complementary cues from both modalities. Evaluations of leading omni‑modal LLMs show significant performance gaps, with many models achieving word error rates above 100%, underscoring challenges in temporal localization, speaker disambiguation, and cross‑modal synergy.
arXiv:2607. 06831v1 Announce Type: cross Abstract: Speech-to-text alignment means finding the temporal boundaries of each word in the audio.
arXiv:2609.14118v1 Announce Type: cross Abstract: Online understanding of who is talking to the camera wearer is a key capability for egocentric social interaction. However, existing talk-to-me (TTM)...
The paper compares text‑based and feature‑based models for recognizing compound emotions in real‑world videos. It proposes textualizing non‑verbal cues from audio and visual modalities into text to leverage large language models, while feature‑based models directly combine extracted multimodal features. Experiments on the C‑EXPR‑DB dataset show that feature‑based models outperform textualization in the wild, though textual models can excel when rich transcripts are available.
arXiv:2607. 12774v1 Announce Type: cross Abstract: This article presents our results for the 11th Affective Behavior Analysis in-the-Wild (ABAW) competition.
arXiv:2607. 21496v1 Announce Type: cross Abstract: Cognitive impairment (CI) is a growing public health concern.