arXiv AI
Nov 5

Same Words, Different Judgments: How Preferences Vary Across Modalities

The paper reports the first controlled cross‑modal study comparing human and synthetic preference annotations for identical content presented in text and audio. Using intraclass correlation coefficients, the authors find that achieving good agreement within each modality requires about nine raters, but cross‑modal agreement is near chance. The study shows that audio raters use narrower decision thresholds, exhibit less length bias, and focus more on user‑oriented criteria, indicating that evaluation protocols for audio preference data must be designed specifically for that modality.

By Aaron Broukhim, Nadir Weibel, Eshin Jolly
arXiv AI
Sep 18

Reading Emotions in the Token Space: Discriminative Adaptation of SpeechLLMs for Emotion Recognition

The paper introduces a discriminative adaptation for SpeechLLMs that reads the hidden state of the final prompt token via a simple classification head, enabling emotion recognition in a single forward pass without altering the backbone. This approach replaces the generative decoder, which can produce out‑of‑set labels and favor frequent classes, with a controlled comparison between generative and discriminative inference. Experiments on IEMOCAP show improved Macro F1 scores, elimination of hallucinations, and larger gains on realistic ASR transcripts, while revealing that emotion directions encode indirect associations reflecting web‑scale text biases.

By Hasindri Watawana, Sergio Burdisso, Esa\'u Villatoro-Tello, Manjunath K E, Kadri Hacioglu, Petr Motlicek, Andreas Stolcke