arXiv AI
Jun 16
NVMOS: Non-Verbal Vocalization Quality Assessment in Speech
arXiv:2606. 15888v1 Announce Type: cross Abstract: Non-verbal vocalizations (NVs), such as laughter, sighs, and coughs, are important acoustic cues for emotion and intent.
When Does a Laugh Begin? Structured Annotator Disagreement in Temporal Laughter Localization
The study investigates structured disagreement among annotators in temporal laughter localization, revealing that disagreements are not random but exhibit systematic patterns—larger at offsets than onsets, more frequent for chuckles than full laughs, and predictable from event attributes. Re-annotating the SMILE-Temporal benchmark with multiple annotators per video shows that evaluating against a single reference annotation can significantly bias system scores and ranking accuracy. The authors propose a disagreement‑calibrated evaluation using conformally calibrated tolerance bands that better reflect the full annotator distribution.
Related stories
arXiv AI
Jun 6
ProSarc: Prosody-Aware Sarcasm Recognition Framework via Temporal Prosodic Incongruity
arXiv:2606. 06168v1 Announce Type: new Abstract: We present ProSarc, an audio-only framework that detects sarcasm by modelling temporal prosodic incongruity, that is, the mismatch between local prosodic dynamics and the utterance-level emotional baseline.
arXiv Computation and Language
Sep 11
Timing is Everything: Temporal Scaffolding of Semantic Surprise in Humor
The paper introduces the Dual Prediction Violation (DPV) framework to study how timing and semantic surprise interact in humor. Analyzing 828 Chinese stand‑up performances, it finds that temporal features—especially pauses before high‑surprise punchlines—are more predictive of audience appreciation than overall semantic incongruity. The study reframes humor as a temporally scaffolded phenomenon where timing and content coordinate strategically rather than independently.
arXiv AI
Oct 2
Comedic Fool's Gold: Reward Exploits and Countermeasures in Conversational Humor
The paper investigates automated reward systems for training language models in conversational humor, examining how reward exploits can undermine intended behavior. It evaluates two reward approaches—an embedding-based surprise reward and an audience-model laughter prediction—showing that each can be tricked by word shuffling or laughter cues, respectively. Countermeasures such as fluency filtering and cue normalization mitigate some attacks but also risk rejecting genuine witty responses, highlighting the difficulty of designing robust rewards.
arXiv AI
Jun 2
v-HUB: A Benchmark for Video Humor Understanding from Vision and Sound
arXiv:2509. 25773v3 Announce Type: replace-cross Abstract: AI models capable of comprehending humor hold real-world promise -- for example, enhancing engagement in human-machine interactions.
arXiv Computer Vision
Sep 16
Predicting Human Disagreement for Calibrated Dynamic Facial Expression Recognition
The paper introduces a disagreement‑aware dynamic facial expression recognition framework that directly learns from raw annotator vote vectors using a Dirichlet‑Multinomial likelihood, preserving both predictive mean and scale‑sensitive supervision. It adds an ambiguity head to estimate annotation entropy for unseen clips and employs a Chow‑style reject rule that integrates ambiguity, vacuity, temporal instability, and input quality for selective prediction. On the DFEW benchmark, the method maintains recognition accuracy while cutting expected calibration error by 30 % and area‑under‑risk‑curve by 15 %, with predicted ambiguity correlating 0.52 (Spearman) with true annotation entropy, and these gains transfer to FERV39k and hold under identity‑ and movie‑disjoint splits.
arXiv AI
Aug 18
Bias-Corrected Ceilings of Emotion Predictability from Human Label Variation Based on Instance-Level Fano Bounds
arXiv:2608. 15619v1 Announce Type: new Abstract: Emotion recognition from text keeps improving on benchmarks, yet whether an accuracy ceiling has been reached is seldom asked with discipline.
arXiv Computation and Language
Nov 4
The Structure of Disagreement in Emotion Annotation
The paper investigates the patterns of disagreement in emotion annotation by analyzing multilabel annotations from 1,677 sentence-level units of Hemingway’s *The Old Man and the Sea* and comparing them with individual-level annotations from BRIGHTER and GoEmotions. It identifies three recurring disagreement structures: systematic overlap of emotion categories, differences in label breadth and boundary treatment, and sparse participation or abstention that mimic disagreement. The study shows that hard aggregation obscures these patterns, while annotator vote-share representations preserve only marginal frequencies, and it proposes a vocabulary for describing disagreement structure.
arXiv Computer Vision
Oct 2
Align Then Reason: A Multimodal Lip-Sync Judge for Dubbing
The paper introduces Align Then Reason (ATR), a multilingual lip‑sync judge that aligns frame‑level lip representations with phonetic units of a candidate text line and then uses a language model to evaluate both content and timing. ATR achieves significant improvements over existing baselines on a seven‑language benchmark, with mean AUC gains of up to 59.4% for 2B reasoners and similar gains across other LLM families. The method also transfers well to unseen languages and outperforms lip‑reading baselines on real dubbing tasks such as dub‑line reranking and script‑to‑clip assignment.
arXiv AI
Sep 18
AVTrace: Diagnosing Audio-Visual Temporal Reasoning in Omni Models
AVTrace is a diagnostic suite designed to evaluate audio‑visual temporal reasoning in omni models. It covers tasks such as onset and span grounding, synchronization, next‑step prediction, cross‑modal localization, chain parsing, and event‑conditioned comprehension, providing 34,114 training examples and balanced development and test splits. Five open omni models were tested, all scoring below the majority‑label baseline on synchronization verification and showing low performance on chain parsing and event‑conditioned tasks, while parameter‑efficient temporal post‑training improved some metrics.
arXiv Computation and Language
Nov 4
Can Large Language Models Capture Video Game Engagement?
The paper investigates whether out‑of‑the‑box pretrained Large Language Models (LLMs) can detect human affect by predicting continuous engagement annotations from video game footage. Using 80 minutes of annotated first‑person shooter gameplay from the GameVibe corpus, the authors run over 4,800 experiments to assess how LLM architecture, size, input modality, prompting strategy, and ground‑truth processing affect performance. Results show that while LLMs can outperform traditional machine‑learning baselines, they generally lag behind human‑provided continuous experience annotations, and the study explores reasons for performance variability and outlines directions for future automated emotion labeling.
arXiv AI
Aug 24
Do SpeechLMs Hear Their Own Opinions? Diagnosing and Mitigating Previous-Belief Contamination in Streaming Emotion Understanding
The paper investigates how streaming emotion recognition models can be misled by their own prior predictions, a problem termed previous-belief contamination (PBC). Using a counterfactual diagnostic on CREMA-D-Stream, the authors show that feeding a model’s previous emotion label into its current prediction can drastically lower accuracy and flip many predictions, with the effect varying by label. To mitigate PBC, they propose EmoUpdate, a training‑free framework that isolates current audio perception from historical context through a prior‑blind firewall, a causal belief filter, and a decontamination operator, achieving significant gains across multiple SpeechLMs and benchmarks.