arXiv:2610.10145v1 Announce Type: new Abstract: We present InterView-C, a German multimodal corpus of 27 survey interviews conducted entirely in virtual reality, with both interlocutors represented b...
The paper introduces ReACT‑TTS, a two‑stage framework that uses a one‑second sequence of listener facial reactions to plan the emotion and prosody of a conversational utterance before speech synthesis. Experiments on the MELD dataset show that temporally conditioned models outperform text‑only baselines in macro‑F1 and VAD concordance, while accuracy remains unchanged. A contextual‑appropriateness study with speech researchers indicates a strong preference for temporally conditioned responses, and the approach is integrated with a Grad‑TTS backbone for end‑to‑end speech generation.
arXiv:2609.14666v1 Announce Type: cross Abstract: Turn-taking is a fundamental component of spoken interaction, and while humans naturally rely on both verbal and non-verbal signals, dialogue systems...
arXiv:2608.22731v1 Announce Type: new Abstract: Nonverbal behavior generation systems for virtual agents often take an utterance as input and generate nonverbal behaviors that emphasize or illustrate...
The paper introduces Omni Demand Understanding (ODU), a benchmark designed to test whether multimodal models can infer a user's underlying demand from complex audio‑visual interactions. ODU requires models to detect the presence of a demand and infer intent using multimodal and conversational context, evaluated across single‑turn and multi‑turn scenarios. The authors built ODU‑Bench through a taxonomy‑guided approach, agentic video generation, and human‑recorded interactions, and found that even top models like Gemini 3.1 Pro recover only 44.7% of key information, with many models exhibiting high false‑trigger rates.
arXiv:2608.20905v1 Announce Type: new Abstract: Face-to-face audiovisual interaction is central to human communication, conveying rich emotional and social cues. However, existing multimodal dialogue...
arXiv:2603. 16859v2 Announce Type: replace Abstract: Omni-modal large language models (OLMs) redefine human-machine interaction by natively integrating audio, vision, and text.
The paper introduces an automated pipeline that extracts conversational turns and backchannels from separate-channel recordings of spontaneous dyadic dialogue, combining voice activity detection, channel-energy filtering, temporal merging, automatic speech recognition, and context-based post‑processing. Evaluated on 99 ten‑minute Danish conversations, the system achieved F1 scores around 0.62 for both turns and backchannels, with median onset/offset errors of roughly 0.15–0.18 s. Performance was consistent across normal and asymmetric listening conditions, and a case study showed the pipeline’s outputs were less variable than human annotations, supporting its use as a reliable first‑pass annotation tool in semi‑automated workflows.
arXiv:2609.14118v1 Announce Type: cross Abstract: Online understanding of who is talking to the camera wearer is a key capability for egocentric social interaction. However, existing talk-to-me (TTM)...
arXiv:2610.15129v1 Announce Type: new Abstract: Human social perception is incremental: we infer intent before an utterance is complete, revise our interpretation as new cues arrive, and attend to sp...
The paper introduces CLEAR, a framework for Multimodal Emotion Recognition in Conversation that couples contextual representations with local discriminative cues. It replaces emotion label generation with continuous internal representations from a vision‑language model and trains a classifier on these representations. Sample‑adaptive residual coupling is used to integrate textual and acoustic evidence from the target utterance into the contextual representation, improving emotion recognition performance on IEMOCAP and MELD datasets.
arXiv:2606. 15888v1 Announce Type: cross Abstract: Non-verbal vocalizations (NVs), such as laughter, sighs, and coughs, are important acoustic cues for emotion and intent.