X-AuT is a progressive framework for compressing the audio encoder of speech large language models. It selects layer combinations via short behavioral probes and restores performance through representation alignment, cross‑scale distillation, scheduled student‑policy supervision, and LoRA finetuning, while keeping the language‑model backbone frozen. On ten Chinese–English benchmarks, reducing Qwen3‑ASR‑0.6B’s encoder from 18 to 16 layers lowers macro‑average error from 5.61% to 5.27%, and a 14‑layer model achieves 5.75% error with 20.7% fewer parameters.
Pruning large pre-trained transformer-based ASR models such as OpenAI's Whisper has seen great adoption, as pruning the decoder led to significant end-to-end transcription speedups. For instance, the...
The paper introduces a method to prune six layers from the encoder of OpenAI’s Whisper ASR model, reducing the encoder stack by 18.5% without requiring custom inference code. Layers are selected based on their minimal impact on Word Error Rate when removed. After pruning, the model’s WER rises from 18.2% to 21.9%, but distillation with unlabeled monolingual speech data lowers it to 20.1%. "whyItMatters":"The approach offers a straightforward way to accelerate Whisper inference by simplifying the encoder while maintaining acceptable accuracy, and the released code and model enable immediate adoption by the community."
arXiv:2603. 05121v2 Announce Type: replace-cross Abstract: Speech Large Language Models route speech encoder representations into an LLM decoder that typically accounts for over 90% of total parameters.
arXiv:2605.23954v2 Announce Type: replace-cross Abstract: Large Audio Language Models (LALMs) remain vulnerable to acoustic noise, which can obscure task-relevant evidence and produce unreliable resp...
arXiv:2608. 08569v1 Announce Type: new Abstract: Recent advancements in Speech Large Language Models have demonstrated remarkable capabilities in understanding complex audio tasks.
arXiv:2606. 01016v1 Announce Type: cross Abstract: While End-to-End (E2E) Speech-Large Language Models (Speech-LLMs) are rapidly evolving, their evaluation methodologies remain limited to the era of simple transcription.
The Qwen-Audio-3.0-ASR Technical Report introduces a Mixture-of-Experts large language model-based automatic speech recognition system that addresses real‑world production challenges such as regional dialects, dynamic entities, hotwords, long‑range context, and disfluent speech. Built on the Qwen backbone and trained on tens of millions of hours of speech data, it supports transcription in 30 languages and 16 Chinese dialects, and offers industry‑domain entity recognition, hierarchical hotword customization, single‑pass polishing, and long‑audio contextual modeling. A streaming variant, Qwen-Audio-3.0-ASR-Streaming, is also presented for low‑latency applications, with evaluations showing state‑of‑the‑art performance against leading commercial systems.
UniAE-MoE is a unified audio encoder that uses a Mixture-of-Experts architecture to model cross‑domain audio representations. It integrates encoder components from Qwen2‑Audio and Audio‑Flamingo 3, enhances them with SwiGLU and shared experts, and applies a two‑stage instruction‑tuning strategy along with task‑specific data scaling. The model achieves state‑of‑the‑art results on the XARES‑LLM benchmark (0.802) and tops the Interspeech 2026 Audio Encoder Capability Challenge, demonstrating strong generalization across speech, music, and general audio tasks.
arXiv:2508. 07048v2 Announce Type: replace-cross Abstract: Autoregressive (AR) encoder-decoder models dominate high-quality multilingual ASR, but their left-to-right decoders make inference latency scale with transcript length.
arXiv:2609.05871v1 Announce Type: cross Abstract: Audio-conditioned language models often underuse acoustic cues such as prosody, emotion, and non-speech sounds, raising the question of whether ASR-s...
arXiv:2601.06086v2 Announce Type: replace Abstract: Extending large language models (LLMs) to the speech domain has recently gained significant attention. A typical approach connects a pretrained LLM...
The paper revisits large‑language‑model (LLM) based automatic speech recognition (ASR) from an entropy‑allocation perspective, introducing three metrics to analyze how entropy reduction is distributed between the speech encoder and the LLM. It proposes a multi‑stage training strategy that addresses entropy‑allocation inefficiencies, including a redesigned pretraining phase to reduce the speech‑text modality gap and an iterative asynchronous fine‑tuning stage to maintain functional decoupling and limit encoder drift. Experiments on Mandarin and English benchmarks demonstrate competitive performance with state‑of‑the‑art models using only 2.3 B parameters, while also reducing hallucinations through the decoupling‑oriented design.