Audio-Visual Turn-taking Prediction in Cocktail Party Scenarios

The paper evaluates audio‑visual predictive turn‑taking models trained on clean data when applied to a noisy cocktail‑party scenario derived from the AVCocktail dataset. Results show a consistent performance drop—up to 38% relative in weighted F1—across both audio and visual modalities, with fine‑tuning improving robustness but varying by modality and pre‑training data size. The study highlights differing generalisation and adaptation abilities of audio versus visual inputs and underscores the need for robust modelling strategies in noisy human interactions.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

• fine-tuning
• multimodal
• benchmarks