arXiv AI

Reading Emotions in the Token Space: Discriminative Adaptation of SpeechLLMs for Emotion Recognition

The paper introduces a discriminative adaptation for SpeechLLMs that reads the hidden state of the final prompt token via a simple classification head, enabling emotion recognition in a single forward pass without altering the backbone. This approach replaces the generative decoder, which can produce out‑of‑set labels and favor frequent classes, with a controlled comparison between generative and discriminative inference. Experiments on IEMOCAP show improved Macro F1 scores, elimination of hallucinations, and larger gains on realistic ASR transcripts, while revealing that emotion directions encode indirect associations reflecting web‑scale text biases.

arXiv Computation and Language
Nov 2

Incongruence-Aware Evaluation of Multimodal Speech Emotion Recognition in Audio Large Language Models

The paper introduces an incongruence score that quantifies disagreement between a speaker’s verbal content and vocal expression in a shared valence‑arousal‑dominance space, validated against human judgments. Using this metric, the authors evaluate three audio‑capable large language models across acted, conversational, and spontaneous speech datasets, finding that higher incongruence consistently increases error rates when audio is present. The study also shows that supervised classifiers behave oppositely, suggesting the challenge lies in how models fuse multimodal evidence rather than in the signal itself, and that the score remains predictive even without an emotion lexicon.

By Fengji Jing, Guy Laban
arXiv Computation and Language
Oct 16

Is Text All You Need? Text as a Universal Information Bottleneck for Speech LLMs

The paper introduces Convex Gate (C‑Gate), a bridge that maps each speech frame to a convex combination of frozen token embeddings, enabling a single LLM decoder to process both speech and text. Using a Whisper‑large‑v3 encoder and a partially adapted Qwen2.5‑7B‑Instruct decoder, the authors jointly train on speech recognition, emotion recognition, and spoken reasoning, achieving accurate transcription and emotion detection while maintaining strong performance on text‑only benchmarks. Analyses reveal that the fixed text embeddings can support both semantic and acoustic readouts through continuous, speech‑conditioned mixtures.

By Ming-Hao Hsu, Yuxuan Hu, Shujie Liu, Jinyu Li, Yan Lu, Zhizheng Wu
arXiv Computation and Language
Sep 30

HPRO: Hierarchical Progressive Reward Optimization via Preference Extraction for Emotional Text-to-Speech

arXiv:2606.28249v2 Announce Type: replace-cross Abstract: Recently, Large Language Model (LLM)-based Text-to-Speech (TTS) models have achieved remarkable naturalness. However, the standard Supervised...

By Sihang Nie, Xiaofen Xing, Rui Xing, Haoming Li, Ruitong Xiao, Jingyuan Xing, Baiji Liu, Xiangmin Xu