The paper revisits large‑language‑model (LLM) based automatic speech recognition (ASR) from an entropy‑allocation perspective, introducing three metrics to analyze how entropy reduction is distributed between the speech encoder and the LLM. It proposes a multi‑stage training strategy that addresses entropy‑allocation inefficiencies, including a redesigned pretraining phase to reduce the speech‑text modality gap and an iterative asynchronous fine‑tuning stage to maintain functional decoupling and limit encoder drift. Experiments on Mandarin and English benchmarks demonstrate competitive performance with state‑of‑the‑art models using only 2.3 B parameters, while also reducing hallucinations through the decoupling‑oriented design.
arXiv:2608. 00722v1 Announce Type: cross Abstract: Language model-based text-to-speech (LM-based TTS) remains vulnerable to speech hallucinations that deviate from the target text.
arXiv:2609.17981v1 Announce Type: cross Abstract: Speech Large Language Models (Speech-LLMs), typically built from a pre-trained speech encoder, a modality projector, and an LLM fine-tuned with Low-R...
Speech Large Language Models (Speech-LLMs), typically built from a pre-trained speech encoder, a modality projector, and an LLM fine-tuned with Low-Rank Adapters (LoRA), have shown strong Automatic Sp...
arXiv:2603. 05121v2 Announce Type: replace-cross Abstract: Speech Large Language Models route speech encoder representations into an LLM decoder that typically accounts for over 90% of total parameters.
arXiv:2606. 09234v1 Announce Type: cross Abstract: Recent state-of-the-art (SOTA) text-to-speech (TTS) systems typically adopt a cascaded pipeline consisting of a speech tokenizer, an autoregressive large language model (LLM), and a diffusion based flow-matching (FM) model, with these components trained independently.
The paper introduces Convex Gate (C‑Gate), a bridge that maps each speech frame to a convex combination of frozen token embeddings, enabling a single LLM decoder to process both speech and text. Using a Whisper‑large‑v3 encoder and a partially adapted Qwen2.5‑7B‑Instruct decoder, the authors jointly train on speech recognition, emotion recognition, and spoken reasoning, achieving accurate transcription and emotion detection while maintaining strong performance on text‑only benchmarks. Analyses reveal that the fixed text embeddings can support both semantic and acoustic readouts through continuous, speech‑conditioned mixtures.
arXiv:2606. 00819v1 Announce Type: new Abstract: Large Language Models (LLMs) have achieved strong performance across diverse natural language tasks, yet their outputs often suffer from hallucinations -- content that is misaligned with factual information.
The paper introduces a new target‑speaker unlearning task for automatic speech recognition (TSU‑ASR) that allows certain speakers to opt out of transcription while still indicating their presence. A lightweight Enrollment‑Conditioned Gating (ECG) module is added to a frozen dual‑stream speech LLM, enabling dynamic unlearning of new opt‑out speakers during inference. Experiments on AMI and AliMeeting datasets show significant drops in transcription accuracy for opt‑out speakers while preserving performance for retained speakers.
arXiv:2610.08063v1 Announce Type: cross Abstract: This paper presents the HINTT system submitted to the 2nd Challenge and Workshop on Multilingual Conversational Speech Language Model (MLC-SLM). We a...
arXiv:2608. 08569v1 Announce Type: new Abstract: Recent advancements in Speech Large Language Models have demonstrated remarkable capabilities in understanding complex audio tasks.
The paper introduces Dynamic Alignment Compensation (DAC), a training‑free inference‑time technique designed to reduce hallucinations in Large Vision‑Language Models (LVLMs). DAC monitors cross‑modal representation drift across decoder layers and generation steps, applying lightweight residual compensation through Layer‑wise Semantic Compensation and Sequential Semantic Correction. Experiments on nine multimodal benchmarks across various LVLM backbones demonstrate that DAC consistently lowers hallucination rates while preserving overall performance.
Conversational ASR for lower-resource languages and niche domains is limited by the scarcity of domain-matched multi-speaker training data. We propose an augmentation pipeline that generates scenario-level dialogues with participant metadata, maps speaker attributes to TTS voice profiles, and assembles synthesized utterances into speaker-aware simulated conversations.