← Back to all news
arXiv Computation and Language By Long-Vu Hoang, Naomi Harte

Audio-Visual Turn-taking Prediction in Cocktail Party Scenarios

Read the original on arXiv Computation and Language →

The paper evaluates audio‑visual predictive turn‑taking models trained on clean data when applied to a noisy cocktail‑party scenario derived from the AVCocktail dataset. Results show a consistent performance drop—up to 38% relative in weighted F1—across both audio and visual modalities, with fine‑tuning improving robustness but varying by modality and pre‑training data size. The study highlights differing generalisation and adaptation abilities of audio versus visual inputs and underscores the need for robust modelling strategies in noisy human interactions.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv Computation and Language
Aug 27

TurnBench: A Multi-Domain Benchmark for Turn-Taking Dynamics in Spoken Dialogue

TurnBench is a new multi‑domain benchmark for evaluating turn‑taking dynamics in spoken dialogue. It comprises a 30‑hour hand‑labeled corpus of dyadic human conversations, a standardized evaluation protocol for end‑of‑turn and interruption detection, and covers six distinct interaction styles with triple annotation. The benchmark also provides a 104‑hour training set, a public leaderboard, and an interactive dataset viewer at https://turnbench.sesame.com.

By Freeman Jiang, Ramon Sanabria, Soham Deshmukh, Bandhav Veluri, Simon Michael Vuch Williams, Elliott K. Suen, Garreth Lee, Kevin Yoonho Choi, Takuya Umeki, Riku Kubo, Sathvik Udupa, Chien-yu Huang, Shih-Yun Shan Kuan, Zhuoyan Tao, Satyapriya Krishna, Sefik Emre Eskimez, Yu Tsao, Hung-yi Lee, Shinji Watanabe
benchmarks
More like this →
arXiv Machine Learning
Oct 16

CIViC: Evidence Gates for Audio-Conditioned Visual Prediction on Proxy Tasks

CIViC (Conditional Input and Visual Interpretation Checks) is a four‑gate protocol designed to link interpretation to intervention, held‑out task comparison, or endpoint control in audio‑conditioned visual prediction. The authors evaluate CIViC on three proxy tasks—actor‑disjoint face‑raster tests, a synthetic audio‑feature mixture, and source‑video‑disjoint SpeakerVid—showing that different gates reveal distinct effects of audio manipulation on visual outputs. Across these tasks, CIViC demonstrates that output activation, held‑out error, and endpoint controls can support varied interpretations of audio influence on visual predictions.

By Mengzhe Geng, Max W. Y. Lam, Yuewen Cao, Shoukang Hu, Jing Xu, Marc Tessier, Xu Li
More like this →