The doctoral research investigates Audio Emotion Recognition (AER) for individuals with atypical hearing, specifically focusing on auditory hypersensitivity in people with autism. It proposes leveraging affective insights from acoustic traits and aims to generalize emotional responses from limited annotated data. The study begins by fine‑tuning the Contrastive Language‑Audio Pretraining (CLAP) foundation model using low‑rank adaptation (LoRA) on a valence and arousal dataset collected from neurotypical listeners.
arXiv:2608.28932v1 Announce Type: new Abstract: Voice products increasingly need affective cues that are present in speech but absent from transcripts. We introduce VocalAffectBench, a public, test-o...
The paper introduces an LLM-based framework for continuous dimensional emotion evaluation in multimodal dialogue, combining discrete emotion recognition with Valence-Arousal-Dominance (VAD) assessment on the IEMOCAP dataset. It incorporates acoustic cues as natural language descriptions via the SpeechCueLLM approach and evaluates six models from the LLaMA, GPT, and Qwen families using zero-shot, few-shot, and LoRA fine-tuning. LoRA-fine-tuned LLaMA models outperform prompt-engineered GPT models, achieving a new state-of-the-art Valence CCC of 0.7822, and ablation studies show that textual audio descriptions significantly benefit smaller models. "whyItMatters":"The study demonstrates that domain adaptation through fine-tuning can surpass larger GPT models in multimodal emotion evaluation, highlighting the importance of tailored training for emotion recognition tasks."
arXiv:2606. 07293v1 Announce Type: cross Abstract: Speech Emotion Conversion (SEC) aims to transform the emotion of a source utterance into a target emotion while preserving content and speaker identity.
arXiv:2608. 05165v1 Announce Type: cross Abstract: Speech Emotion Recognition (SER) in low-resource languages remains a challenging problem due to limited labeled data.
arXiv:2606. 07309v1 Announce Type: cross Abstract: Instruction-following audio language models (ALMs) can be augmented with explicit acoustic cues, yet it remains unclear whether such cues are used in a grounded way when the raw audio is already available.
arXiv:2607. 11801v1 Announce Type: cross Abstract: Large audio-language models (LALMs) often underperform on fine-grained, non-semantic attributes of speech, such as a speaker's emotion, despite strong performance on speech content.
The paper introduces an incongruence score that quantifies disagreement between a speaker’s verbal content and vocal expression in a shared valence‑arousal‑dominance space, validated against human judgments. Using this metric, the authors evaluate three audio‑capable large language models across acted, conversational, and spontaneous speech datasets, finding that higher incongruence consistently increases error rates when audio is present. The study also shows that supervised classifiers behave oppositely, suggesting the challenge lies in how models fuse multimodal evidence rather than in the signal itself, and that the score remains predictive even without an emotion lexicon.
EmoNet is a lightweight attention‑based CNN framework designed for speech emotion recognition across seven categories. It combines a four‑block depthwise separable CNN encoder with squeeze‑and‑excitation attention, residual connections, a Bidirectional LSTM, and multi‑head attention pooling, using a 134‑channel acoustic feature set and data augmentation to enhance robustness. On a combined TESS, RAVDESS, and SAVEE dataset, EmoNet achieves 96.75% test accuracy with only 4.11 million parameters, outperforming a reproduced baseline and competing with larger state‑of‑the‑art models.
arXiv:2606. 00670v1 Announce Type: cross Abstract: Face-to-face speech comprehension is inherently multimodal, integrating acoustic signals with visible articulation, facial expression, head motion, and other socially relevant cues.
arXiv:2507. 07046v3 Announce Type: replace-cross Abstract: Nowadays, speech emotion recognition (SER) plays a vital role in the field of human-computer interaction (HCI) and the evolution of artificial intelligence (AI).
The paper introduces ACERT, a module that incorporates a flexible-length window of conversational context to enhance Speech Emotion Recognition (SER). By capturing emotional evolution across utterances, ACERT outperforms state‑of‑the‑art methods on IEMOCAP, sets a new context‑aware benchmark on SAFE, and achieves strong results on MELD. Ablation studies attribute ACERT’s improvements to emotional and conversational continuity rather than speaker identity or acoustic conditions.
arXiv:2606. 03359v1 Announce Type: cross Abstract: Speech emotion recognition is an important component of modern human-computer interaction systems.