Speech Large Language Models (Speech-LLMs), typically built from a pre-trained speech encoder, a modality projector, and an LLM fine-tuned with Low-Rank Adapters (LoRA), have shown strong Automatic Sp...
arXiv:2604.06487v2 Announce Type: replace Abstract: Conventional end-to-end automatic speech recognition (ASR) systems rely on paired speech-text data for domain adaptation. Recent LLM-based ASR arch...
arXiv:2509.10452v3 Announce Type: replace-cross Abstract: Pretrained automatic speech recognition (ASR) models such as Whisper perform well but still need domain adaptation to handle unseen parlance....
arXiv:2609.15743v1 Announce Type: new Abstract: Automatic speech recognition (ASR) systems, trained on paired speech-text data, have been improved by leveraging language models (LMs) trained on text-...
The paper revisits large‑language‑model (LLM) based automatic speech recognition (ASR) from an entropy‑allocation perspective, introducing three metrics to analyze how entropy reduction is distributed between the speech encoder and the LLM. It proposes a multi‑stage training strategy that addresses entropy‑allocation inefficiencies, including a redesigned pretraining phase to reduce the speech‑text modality gap and an iterative asynchronous fine‑tuning stage to maintain functional decoupling and limit encoder drift. Experiments on Mandarin and English benchmarks demonstrate competitive performance with state‑of‑the‑art models using only 2.3 B parameters, while also reducing hallucinations through the decoupling‑oriented design.
The paper introduces ABX-Accent, a benchmark built on the AESRC dataset that evaluates how well representation learning models adapt to 10 different English accents with less than 10 hours of unlabeled data per accent. It adapts the Zero Resources Challenge ABX metrics for each accent and demonstrates a baseline using adaptive domain normalization to fine‑tune a Contrastive Predictive Coding model, achieving a 23.6% relative improvement on across‑speaker ABX scores compared to non‑adapted models. The dataset and evaluation metrics will be released publicly after the paper is accepted.
The paper introduces ABX-Accent, a benchmark for evaluating how well unsupervised speech representation learning models adapt to new accents. It uses the AESRC dataset with 10 English accents, each providing less than 10 hours of unlabeled data, and adapts the Zero Resources Challenge ABX metrics to these accents. A baseline model employing adaptive domain normalization to fine‑tune a Contrastive Predictive Coding model shows a 23.6% relative improvement on across‑speaker ABX scores compared to non‑adapted models.
arXiv:2609.23825v1 Announce Type: new Abstract: We present a comprehensive benchmark of Federated Learning (FL) for multilingual Automatic Speech Recognition (ASR), evaluating four Speech-LLM archite...
arXiv:2601. 18904v3 Announce Type: replace-cross Abstract: Generative AI for speech and audio is increasingly expected to serve users across languages, cultures, and communities, yet current auditory Large Language Models (LLMs) are still largely trained and evaluated on high-resource data.
The paper introduces AZeroS, a speech‑adapted large language model that uses Self‑Generated Instruction‑Free Tuning (SIFT) to avoid task‑specific instruction data. By training only two lightweight projection modules on 25,000 hours of speech‑text pairs and 3,000 hours of paralinguistic labels, AZeroS achieves state‑of‑the‑art results on semantic and paralinguistic benchmarks while keeping the core LLM and audio encoder frozen.
The paper introduces asymmetric classifier‑free guidance (CFG) for target‑speaker ASR using Whisper, where a speaker‑conditioned branch predicts the target transcript and a speaker‑unconditioned branch predicts serialized multi‑speaker transcripts. CFG modulates the influence of speaker conditioning during decoding via a single guidance scale, which is first set globally on development data and then refined per utterance by a lightweight encoder‑based predictor while keeping the recognition model fixed. The resulting system yields up to 21.8% relative WER reduction over a condition‑only baseline and 5.6% over standard conditional decoding under domain shifts.
Connecting a pre-trained speech encoder to a Large Language Model (LLM) is the standard architecture for building Speech LLMs. However, a structural misalignment exists between the encoder and the LLM.