ConvexLLM introduces a C‑Gate interface that represents audio as continuous convex combinations of frozen text embeddings, aiming to preserve pretrained text capabilities while incorporating acoustic information. The model achieves a VoiceBench aggregate score of 75.36 and an MMAU Test‑mini accuracy of 62.50%, outperforming MLP and Q‑Former interfaces on spoken knowledge and reasoning tasks. Compared to a 2B‑parameter decoder, C‑Gate delivers higher spoken‑language performance with comparable text‑task results and task‑dependent tradeoffs in transcription and audio understanding.
By Ming-Hao Hsu, Shujie Liu, Yuxuan Hu, Yuxiang Wang, Yingda Shen, Jinyu Li, Zhizheng Wu
arXiv:2606. 09234v1 Announce Type: cross Abstract: Recent state-of-the-art (SOTA) text-to-speech (TTS) systems typically adopt a cascaded pipeline consisting of a speech tokenizer, an autoregressive large language model (LLM), and a diffusion based flow-matching (FM) model, with these components trained independently.
By Changfeng Gao, Yong Ren, Jun Yuan, Ye Bai, Zhao You, ShiDong Shang
arXiv:2607. 05196v1 Announce Type: cross Abstract: Audio intelligence involves understanding, reasoning about, and generating both audio and speech.
By Zhifeng Kong, Sang-gil Lee, Jaehyeon Kim, Boxin Wang, Zihan Liu, Sungwon Kim, Yang Chen, Arushi Goel, Rajarshi Roy, Wenliang Dai, Zhuolin Yang, Yangyi Chen, Dongfu Jiang, Sreyan Ghosh, Tuomas Rintamaki, Andrew Tao, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, Wei Ping
arXiv:2610.08085v1 Announce Type: new
Abstract: Speech-LLMs often exhibit prompt overfitting, where models solely trained on automatic speech recognition (ASR) instruction fail to generalize to new i...
By Hemant Yadav, Sunayana Sitaram, Roger Zimmermann, Rajiv Ratn Shah
The paper introduces Local Prototype Reconstruction (LPR), a lightweight regularizer that encourages bridge embeddings in speech‑to‑LLM systems to stay close to the frozen LLM’s input‑embedding neighbourhoods. By measuring lexical manifold compatibility with a fixed, head‑free, timestamp‑free diagnostic, the authors show that next‑word prediction and sentence‑level contrastive pretraining do not fully capture token‑level lexical compatibility. Experiments on multilingual ASR and speech translation demonstrate that LPR improves transfer, especially for translation and low‑resource adaptation, and that the diagnostic correlates with downstream gains across objectives.
By Xinnian Zhao, Chia-Hua Wu, Pu Wang, Hugo Van Hamme
Audio intelligence involves understanding, reasoning about, and generating both audio and speech. In this work, we introduce Nemotron-Labs-Audex-30B-A3B (Audex), a unified audio-text LLM built on Nemotron-Cascade-2-30B-A3B, a strong text-only MoE LLM.