SeetaPsych v1.0 is an open‑source computer vision toolkit that unifies and extends modules for behavior‑based psychological measurement, focusing on facial image and video analysis. It includes four core modules—emotion analysis, camera‑based heart rate estimation, screen point‑of‑gaze estimation, and scene gaze following—alongside preprocessing tools such as face, landmark, and head detection. The toolkit offers a modular Pipeline/Runner architecture, standardized Python APIs, and an interactive WebUI to enable reproducible, large‑scale research across psychology, behavioral science, and human‑computer interaction.
By Jiabei Zeng, Chiqin Li, Kaizhou Li, Fei Chang, Yong Li, Yuanhao Zhao, Dan Han, Wenqiang Yang, Xilin Chen, Shiguang Shan
Online understanding of who is talking to the camera wearer is a key capability for egocentric social interaction. However, existing talk-to-me (TTM) studies are commonly formulated as offline clip-le...
arXiv:2609.14118v1 Announce Type: cross
Abstract: Online understanding of who is talking to the camera wearer is a key capability for egocentric social interaction. However, existing talk-to-me (TTM)...
By Feiyu Du, Xi He, Jia Li, Yapeng Tian, Weili Wu
HUG‑VIS is a unified multimodal benchmark for human‑centered visual intelligence, comprising 8,400 half‑body videos of 30 professional actors performing 280 emotion‑action prompts in Mandarin. The dataset provides synchronized video, audio, text, and alpha mattes for four tasks—human emotion recognition, video generation, voice cloning, and video matting—allowing evaluation of both open‑ and closed‑source models under a zero‑shot protocol. Results reveal that linguistic cues dominate emotion recognition, visual affect is weakest, and that automatic metrics and human judgments diverge in generation and cloning tasks, while motion‑related boundary fidelity remains a key challenge for matting.
By Fei Ma, Zebang Cheng, Minghui Li, Hongbo Xu, Yuyong Tan, Yihua Shao, Hanling Wang, Zhou Liu, Yuqing Gao, Dong Wang, Long Ma, Laizhong Cui, Nicu Sebe, Qi Tian
Humans possess an innate ability to understand fine-grained interpersonal relationships, which is central to everyday social interactions. Although such reasoning is inherently multimodal, it remains largely unexplored by existing multimodal large language models (MLLMs).
arXiv:2608.20905v1 Announce Type: new
Abstract: Face-to-face audiovisual interaction is central to human communication, conveying rich emotional and social cues. However, existing multimodal dialogue...
By Yi Zheng, Yifan Xu, Yan Zhou, Hejia Chen, Chunyu Qiang, Xiaoqiang Liu, Xiaohan Li, Shenze Huang, Yue Zhang, Guoying Zhao, Pengfei Wan
arXiv:2609.13255v1 Announce Type: new
Abstract: Facial state analysis plays a crucial role in understanding human expressions, psychological modeling, and human computer interaction. Traditional unim...
By Xuri Ge, Tianshuo Zhang, Ruihan Li, Hui Ye, Kaiwen Zheng, Junchen Fu, Da Huo, Joemon M. Jose, Hu Han
arXiv:2607. 23845v1 Announce Type: new Abstract: Other-initiated Self-repair, or in short Other-initiated Repair (OIR), is an essential mechanism in conversational interaction, whereby a recipient signals a problem in speaking, hearing, or understanding, prompting the previous speaker to resolve it.
By Anh Ngo, Nicolas Rollet, Catherine Pelachaud, Chlo\'e Clavel
PIVOTSBench is a benchmark designed to assess multimodal large language models’ ability to reason about fine‑grained interpersonal relationships. It is constructed from Social‑IQ 2.0 and YouTube data and evaluates models on predicting bidirectional relationship dimensions grounded in psychology research. The benchmark also includes auxiliary tasks that test models’ capacity to identify and use critical visual cues, and it examines the impact of visual modalities, social role information, and different prediction settings on model performance.
By Shuxiang Zhang, Yiting Yin, Wenxuan Song, Yuhang Wu, Miao Liu
arXiv:2608.22731v1 Announce Type: new
Abstract: Nonverbal behavior generation systems for virtual agents often take an utterance as input and generate nonverbal behaviors that emphasize or illustrate...
By Parisa Ghanad Torshizi, Stacy Marsella
arXiv:2605.23328v3 Announce Type: replace
Abstract: Emotion Recognition in Conversation is a core component of affective computing, while current sign language emotion datasets primarily focus on iso...
By Yusong Wang, Keyu Mao, Takao Obi, Minghao Shao, Kotaro Funakoshi
Nonverbal behavior generation systems for virtual agents often take an utterance as input and generate nonverbal behaviors that emphasize or illustrate the content of the verbal channel. However, huma...