← Back to all news
arXiv Computation and Language By Ming-Hao Hsu, Shujie Liu, Yuxuan Hu, Yuxiang Wang, Yingda Shen, Jinyu Li, Zhizheng Wu

ConvexLLM: Scaling Convex-Hull Speech Interfaces to General-Purpose Audio Intelligence

Read the original on arXiv Computation and Language →

ConvexLLM introduces a C‑Gate interface that represents audio as continuous convex combinations of frozen text embeddings, aiming to preserve pretrained text capabilities while incorporating acoustic information. The model achieves a VoiceBench aggregate score of 75.36 and an MMAU Test‑mini accuracy of 62.50%, outperforming MLP and Q‑Former interfaces on spoken knowledge and reasoning tasks. Compared to a 2B‑parameter decoder, C‑Gate delivers higher spoken‑language performance with comparable text‑task results and task‑dependent tradeoffs in transcription and audio understanding.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv Computation and Language
6d ago

Is Text All You Need? Text as a Universal Information Bottleneck for Speech LLMs

The paper introduces Convex Gate (C‑Gate), a bridge that maps each speech frame to a convex combination of frozen token embeddings, enabling a single LLM decoder to process both speech and text. Using a Whisper‑large‑v3 encoder and a partially adapted Qwen2.5‑7B‑Instruct decoder, the authors jointly train on speech recognition, emotion recognition, and spoken reasoning, achieving accurate transcription and emotion detection while maintaining strong performance on text‑only benchmarks. Analyses reveal that the fixed text embeddings can support both semantic and acoustic readouts through continuous, speech‑conditioned mixtures.

By Ming-Hao Hsu, Yuxuan Hu, Shujie Liu, Jinyu Li, Yan Lu, Zhizheng Wu
llmsragmultimodalbenchmarks
More like this →
arXiv AI
Jul 7

Unified Audio Intelligence Without Regressing on Text Intelligence

arXiv:2607. 05196v1 Announce Type: cross Abstract: Audio intelligence involves understanding, reasoning about, and generating both audio and speech.

By Zhifeng Kong, Sang-gil Lee, Jaehyeon Kim, Boxin Wang, Zihan Liu, Sungwon Kim, Yang Chen, Arushi Goel, Rajarshi Roy, Wenliang Dai, Zhuolin Yang, Yangyi Chen, Dongfu Jiang, Sreyan Ghosh, Tuomas Rintamaki, Andrew Tao, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, Wei Ping
llmsragagentsreinforcement-learningefficiencymultimodalbenchmarkssafety
More like this →
arXiv AI
Jul 17

RW-Voice-EQ Bench: A Real World Benchmark for Evaluating Voice AI Systems

arXiv:2607. 14846v1 Announce Type: cross Abstract: Current voice AI benchmarks typically evaluate isolated capabilities such as speech intelligibility, word error rate, or text-based dialogue quality, but they rarely test whether systems harness the acoustic information that distinguishes spoken language from its textual representation.

By David Ayllon, Alice Baird, Jeffrey Brooks, Franc Camps-Febrer, Jakub Piotr C{\l}apa, Theo Lebryk, Jens Madsen, Olya Ossipova, Sharath Rao, Hoon Shin, Tigran Soghbatyan, Georg Streich, Rashish Tandon, Panagiotis Tzirakis
agentsmultimodalbenchmarks
More like this →