← Back to all news
arXiv Computation and Language

Can Large Language Models Capture Video Game Engagement?

The paper investigates whether out‑of‑the‑box pretrained Large Language Models (LLMs) can detect human affect by predicting continuous engagement annotations from video game footage. Using 80 minutes of annotated first‑person shooter gameplay from the GameVibe corpus, the authors run over 4,800 experiments to assess how LLM architecture, size, input modality, prompting strategy, and ground‑truth processing affect performance. Results show that while LLMs can outperform traditional machine‑learning baselines, they generally lag behind human‑provided continuous experience annotations, and the study explores reasons for performance variability and outlines directions for future automated emotion labeling.

Related stories

arXiv AI
Oct 2

Do Vision Language Models Understand Human Engagement in Games?

The study investigates whether vision–language models (VLMs) can infer human engagement from gameplay videos using the GameVibe Few‑Shot dataset across nine first‑person shooter games. Three VLMs were tested under six prompting strategies—including zero‑shot, theory‑guided prompts based on Flow, GameFlow, Self‑Determination Theory, and MDA, and retrieval‑augmented prompting—evaluating both pointwise engagement prediction and pairwise prediction of engagement change. Results show that zero‑shot predictions are weak and often do not beat simple majority‑class baselines; retrieval‑augmented prompting improves pointwise prediction in some cases, while pairwise prediction remains difficult, and theory‑guided prompts do not reliably help and may reinforce superficial shortcuts. "whyItMatters":"The findings highlight a perception–understanding gap in current VLMs, indicating that while they can recognize visible gameplay cues, they still struggle to robustly infer human engagement across games."

By Ziyi Wang, Qizan Guo, Rishitosh Singh, Xiyang Hu
llmsragmultimodal
More like this →
arXiv Computer Vision
Sep 3

Video2Reaction: Training Foundation Video Models to Predict Audience Reaction

Video2Reaction is a multimodal dataset that links short movie segments to the emotional reactions of viewers, gathered from social media comments. The dataset models reactions as distributions over categorical emotions, capturing the subjective and ambiguous nature of emotional perception. Experiments show that vision‑language models fine‑tuned with LoRA learn effectively from Video2Reaction and outperform specialized baselines, and that models pre‑fine‑tuned on this dataset transfer well to other emotion prediction tasks.

By Sidong Zhang, Trang Nguyen, Shiv Shankar, Gauri Jagatap, Deepak Chandran, Andrea Fanelli, Madalina Fiterau
llmsfine-tuningmultimodalbenchmarks
More like this →
arXiv AI
Oct 7

Real-Time Generation of Game Video Commentary with Multimodal LLMs: Pause-Aware Decoding Approaches

arXiv:2603.02655v2 Announce Type: replace-cross Abstract: Real-time video commentary generation provides textual descriptions of ongoing events in videos. It supports accessibility and engagement in...

By Anum Afzal, Yuki Saito, Hiroya Takamura, Katsuhito Sudoh, Shinnosuke Takamichi, Graham Neubig, Florian Matthes, Tatsuya Ishigaki
llmsfine-tuningmultimodalbenchmarks
More like this →
arXiv Computer Vision
Oct 28

GameVerse: A Minute-Scale Gameplay Dataset for Long-Horizon Interactive World Modeling

GameVerse is a new minute‑scale gameplay video dataset designed to support long‑horizon interactive world modeling. It contains 69.7K clips totaling 1,655 hours and 349.6 million frames, including 1,358 hours of continuous two‑minute sequences—an order of magnitude longer than prior datasets. Each clip is processed through a unified pipeline that provides scene‑consistent segmentation, quality filtering, camera pose estimation, and event‑based caption annotation, with temporally stable camera trajectories and captions that distinguish static context from dynamic events.

By Kang He, Wenshuo Peng, Chuanhao Li, Zihui Gao, Xiaojie Xu, Zhengyuan Lin, Yuzhe Ding, Donghong Ji, Kaipeng Zhang, Yongtao Ge
computer-vision
More like this →
arXiv Computer Vision
Sep 22

GameHorizon Suite: Multi-Horizon Data and Evaluation in Gameplay

arXiv:2609.25001v1 Announce Type: new Abstract: Modern video games provide a measurable testbed for AI models, combining abilities of visual understanding, instruction decomposition, goal planning, a...

By Yiran Wang, Xingyilang Yin, Junfu Pu, Guangzhi Wang, Kaifeng Li, Mingyu Ouyang, Huiqiang Sun, Lingen Li, Cheng Cheng, Wangbo Yu, Honghao Chen, Xiaodong Cun, Chi-Man Pun, Zhiguo Cao, Ying Shan
benchmarks
More like this →