arXiv Computation and Language October 12, 2026 By Yuan Xie, Jiaqi Song, Guang Qiu, Xianliang Wang, Ming Lei, Jie Gao, Jie Wu

Rethinking Entropy Allocation in LLM-based ASR: Understanding the Dynamics between Speech Encoders and LLMs

Read the original on arXiv Computation and Language →

The paper revisits large‑language‑model (LLM) based automatic speech recognition (ASR) from an entropy‑allocation perspective, introducing three metrics to analyze how entropy reduction is distributed between the speech encoder and the LLM. It proposes a multi‑stage training strategy that addresses entropy‑allocation inefficiencies, including a redesigned pretraining phase to reduce the speech‑text modality gap and an iterative asynchronous fine‑tuning stage to maintain functional decoupling and limit encoder drift. Experiments on Mandarin and English benchmarks demonstrate competitive performance with state‑of‑the‑art models using only 2.3 B parameters, while also reducing hallucinations through the decoupling‑oriented design.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

arXiv Machine Learning

Aug 4

Experience-Calibrated Contrastive Decoding for Mitigating Hallucinations in LM-Based Text-to-Speech

arXiv:2608. 00722v1 Announce Type: cross Abstract: Language model-based text-to-speech (LM-based TTS) remains vulnerable to speech hallucinations that deviate from the target text.

By Chenlin Liu, Minghui Fang, Zhonghao Bi, Zekai Su, Rong Wang, Jiqing Han

llms multimodal safety

arXiv Computation and Language

Sep 17

Encoder Awakening via Adapters: Effective Domain-Adaptive Fine-tuning of Speech-LLMs

arXiv:2609.17981v1 Announce Type: cross Abstract: Speech Large Language Models (Speech-LLMs), typically built from a pre-trained speech encoder, a modality projector, and an LLM fine-tuned with Low-R...

By Mohan Shi, Zilai Wang, Natarajan Balaji Shankar, Kaiyuan Zhang, Eray Eren, Abeer Alwan

llms fine-tuning multimodal benchmarks

Hugging Face Trending Papers

Sep 16

Encoder Awakening via Adapters: Effective Domain-Adaptive Fine-tuning of Speech-LLMs

Speech Large Language Models (Speech-LLMs), typically built from a pre-trained speech encoder, a modality projector, and an LLM fine-tuned with Low-Rank Adapters (LoRA), have shown strong Automatic Sp...

llms fine-tuning multimodal benchmarks

arXiv AI

Jun 29

Measuring the Redundancy of Decoder Layers in SpeechLLMs

arXiv:2603. 05121v2 Announce Type: replace-cross Abstract: Speech Large Language Models route speech encoder representations into an LLM decoder that typically accounts for over 90% of total parameters.

By Adel Moumen, Guangzhi Sun, Philip C Woodland

llms efficiency multimodal

arXiv AI

Jun 9

End-to-End Training for Discrete Token LLM based TTS System

arXiv:2606. 09234v1 Announce Type: cross Abstract: Recent state-of-the-art (SOTA) text-to-speech (TTS) systems typically adopt a cascaded pipeline consisting of a speech tokenizer, an autoregressive large language model (LLM), and a diffusion based flow-matching (FM) model, with these components trained independently.

By Changfeng Gao, Yong Ren, Jun Yuan, Ye Bai, Zhao You, ShiDong Shang

llms diffusion nlp reinforcement-learning multimodal benchmarks

arXiv Computation and Language

6d ago

Is Text All You Need? Text as a Universal Information Bottleneck for Speech LLMs

The paper introduces Convex Gate (C‑Gate), a bridge that maps each speech frame to a convex combination of frozen token embeddings, enabling a single LLM decoder to process both speech and text. Using a Whisper‑large‑v3 encoder and a partially adapted Qwen2.5‑7B‑Instruct decoder, the authors jointly train on speech recognition, emotion recognition, and spoken reasoning, achieving accurate transcription and emotion detection while maintaining strong performance on text‑only benchmarks. Analyses reveal that the fixed text embeddings can support both semantic and acoustic readouts through continuous, speech‑conditioned mixtures.

By Ming-Hao Hsu, Yuxuan Hu, Shujie Liu, Jinyu Li, Yan Lu, Zhizheng Wu

llms rag multimodal benchmarks

One email a morning, machine-written

Related stories

Experience-Calibrated Contrastive Decoding for Mitigating Hallucinations in LM-Based Text-to-Speech

Encoder Awakening via Adapters: Effective Domain-Adaptive Fine-tuning of Speech-LLMs

Encoder Awakening via Adapters: Effective Domain-Adaptive Fine-tuning of Speech-LLMs

Measuring the Redundancy of Decoder Layers in SpeechLLMs

End-to-End Training for Discrete Token LLM based TTS System

Is Text All You Need? Text as a Universal Information Bottleneck for Speech LLMs