Humans possess an innate ability to understand fine-grained interpersonal relationships, which is central to everyday social interactions. Although such reasoning is inherently multimodal, it remains largely unexplored by existing multimodal large language models (MLLMs).
PIVOTSBench is a benchmark designed to assess multimodal large language models’ ability to reason about fine‑grained interpersonal relationships. It is constructed from Social‑IQ 2.0 and YouTube data and evaluates models on predicting bidirectional relationship dimensions grounded in psychology research. The benchmark also includes auxiliary tasks that test models’ capacity to identify and use critical visual cues, and it examines the impact of visual modalities, social role information, and different prediction settings on model performance.
arXiv:2609.14666v1 Announce Type: cross Abstract: Turn-taking is a fundamental component of spoken interaction, and while humans naturally rely on both verbal and non-verbal signals, dialogue systems...
arXiv:2609.16396v1 Announce Type: new Abstract: Negation is typically modeled through its linguistic realization, although spoken interaction is accompanied by tightly coordinated nonverbal behavior....
arXiv:2607. 23845v1 Announce Type: new Abstract: Other-initiated Self-repair, or in short Other-initiated Repair (OIR), is an essential mechanism in conversational interaction, whereby a recipient signals a problem in speaking, hearing, or understanding, prompting the previous speaker to resolve it.
arXiv:2609.16464v1 Announce Type: cross Abstract: The clinical management of autism spectrum disorder (ASD) faces a bottleneck in early screening, mainly because trained specialists are scarce and co...
arXiv:2603. 16859v2 Announce Type: replace Abstract: Omni-modal large language models (OLMs) redefine human-machine interaction by natively integrating audio, vision, and text.
The paper introduces HiRe-DRR, a Hierarchical Multimodal Interaction Reasoning framework for dyadic relationship recognition that organizes evidence across feature level, interaction type, and relationship hierarchy. It decomposes facial, bodily, acoustic, and textual cues into low- and high-level representations, builds homogeneous and heterogeneous cross-participant interactions, and employs a Mixture-of-Interaction Transformer with relation-aware sparse routing. A Hierarchical Conditional Loss provides coarse-to-fine supervision, and experiments on three benchmarks show HiRe-DRR outperforms existing image, video, and multimodal methods, with routing analysis revealing distinct interaction patterns for distant versus close relationships.
Challenging behaviors including aggression, self-injury, and property destruction are observed in 68% of autistic youth and pose risks to youth and caregivers. These episodes are preceded by agitation...
arXiv:2607. 08533v1 Announce Type: new Abstract: Understanding perceptual differences between autistic and neurotypical adults requires behavioral assays that are sensitive, reliable, and mechanistically informative.
arXiv:2609.14118v1 Announce Type: cross Abstract: Online understanding of who is talking to the camera wearer is a key capability for egocentric social interaction. However, existing talk-to-me (TTM)...
arXiv:2608. 12548v1 Announce Type: cross Abstract: Dance imitation integrates motor planning, sensorimotor integration, and social cognition, offering a sensitive framework to characterize motor behavior in autism.