arXiv Computer Vision September 3, 2026 By Sidong Zhang, Trang Nguyen, Shiv Shankar, Gauri Jagatap, Deepak Chandran, Andrea Fanelli, Madalina Fiterau

Video2Reaction: Training Foundation Video Models to Predict Audience Reaction

Read the original on arXiv Computer Vision →

Video2Reaction is a multimodal dataset that links short movie segments to the emotional reactions of viewers, gathered from social media comments. The dataset models reactions as distributions over categorical emotions, capturing the subjective and ambiguous nature of emotional perception. Experiments show that vision‑language models fine‑tuned with LoRA learn effectively from Video2Reaction and outperform specialized baselines, and that models pre‑fine‑tuned on this dataset transfer well to other emotion prediction tasks.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computer Vision.

arXiv Machine Learning

Jul 9

Video2Reaction: Mapping Video to Audience Reaction Distribution in the Wild

arXiv:2607. 06875v1 Announce Type: cross Abstract: Understanding and forecasting audience reactions to video content are crucial for improving content creation, recommendation systems, and media analysis.

By Trang Nguyen, Sidong Zhang, Shiv Shankar, Gauri Jagatap, Deepak Chandran, Andrea Fanelli, Madalina Fiterau

llms agents fine-tuning multimodal benchmarks

Hugging Face Trending Papers

3d ago

What Does the Model See? Representation Validity in Multimodal LLM-Based Video Analysis

Multimodal large language models (MLLMs) allow social scientists to analyze video at scale, but researchers must decide which parts of a video to provide as input. We use the term representation valid...

llms multimodal benchmarks

arXiv Computation and Language

3d ago

Can Large Language Models Capture Video Game Engagement?

The paper investigates whether out‑of‑the‑box pretrained Large Language Models (LLMs) can detect human affect by predicting continuous engagement annotations from video game footage. Using 80 minutes of annotated first‑person shooter gameplay from the GameVibe corpus, the authors run over 4,800 experiments to assess how LLM architecture, size, input modality, prompting strategy, and ground‑truth processing affect performance. Results show that while LLMs can outperform traditional machine‑learning baselines, they generally lag behind human‑provided continuous experience annotations, and the study explores reasons for performance variability and outlines directions for future automated emotion labeling.

By David Melhart, Matthew Barthet, Georgios N. Yannakakis

llms multimodal

arXiv AI

Jul 22

Why Do Vision Language Models Struggle To Recognize Human Emotions?

arXiv:2604. 15280v2 Announce Type: replace-cross Abstract: Understanding emotions is a fundamental ability for intelligent systems to be able to interact with humans.

By Madhav Agarwal, Sotirios A. Tsaftaris, Laura Sevilla-Lara, Steven McDonagh

llms multimodal safety

arXiv AI

Oct 28

CLEAR: Coupling Local Evidence with Affective Representations for Multimodal Emotion Recognition in Conversation

The paper introduces CLEAR, a framework for Multimodal Emotion Recognition in Conversation that couples contextual representations with local discriminative cues. It replaces emotion label generation with continuous internal representations from a vision‑language model and trains a classifier on these representations. Sample‑adaptive residual coupling is used to integrate textual and acoustic evidence from the target utterance into the contextual representation, improving emotion recognition performance on IEMOCAP and MELD datasets.

By Linan Zhu, Zihao Zhai, Mengfei Xu, Xiao Han, Yuqian Fu, Xiangfan Chen, Zihang Zhang

llms multimodal

arXiv Computation and Language

Sep 4

Chehre: An Emoji-Prompted Dataset to Explore Perceptual Flexibility in Video Language Models

Chehre is an emoji‑prompted video dataset designed to study perceptual flexibility in video language models. It contains 2,111 videos of 203 participants expressing 40 facial emojis, with each video annotated by about 30 perceivers, yielding 1,242 annotators in total. The dataset introduces a new task—distributional expression recognition—that evaluates a model’s ability to reproduce the variation seen in human annotations, and shows that persona prompting can shift model perception to better match human variability.

By Bita Azari, Zoe Stanley, Avneet Batra, Poorvi Bhatia, Hali Kil, Manolis Savva, Angelica Lim

llms safety