arXiv Computation and Language

Inference-Time Target Speaker Unlearning in LLM-Based Automatic Speech Recognition

The paper introduces a new target‑speaker unlearning task for automatic speech recognition (TSU‑ASR) that allows certain speakers to opt out of transcription while still indicating their presence. A lightweight Enrollment‑Conditioned Gating (ECG) module is added to a frozen dual‑stream speech LLM, enabling dynamic unlearning of new opt‑out speakers during inference. Experiments on AMI and AliMeeting datasets show significant drops in transcription accuracy for opt‑out speakers while preserving performance for retained speakers.

arXiv AI
Jun 18

Speaker Verification with Speech-Aware LLMs: Evaluation and Augmentation

arXiv:2603. 10827v2 Announce Type: replace-cross Abstract: Speech-aware large language models (LLMs) can accept speech inputs, yet their training objectives largely emphasize linguistic content or specific fields such as emotions or the speaker's gender, leaving it unclear whether they encode speaker identity.

By Thomas Thebaud, Yuzhe Wang, Laureano Moro-Velazquez, Jesus Villalba-Lopez, Najim Dehak
arXiv Computation and Language
Sep 21

Transsion's Speaker-Attributed Multilingual ASR System for the MLC-SLM 2026 Challenge

The paper describes Transsion Speech Team’s submission to Task 1 of the MLC‑SLM 2026 Challenge, aiming at speaker‑attributed transcription for multilingual conversational speech. Their cascaded framework includes a DiariZen‑based speaker diarization module, a Qwen3‑Omni‑based long‑form multilingual ASR module with CTC alignment for precise timestamps, and a fusion module that merges diarization and transcription outputs into speaker‑attributed STM results. On the official evaluation set, the system achieved a tcpMER of 15.41% and secured second place among all participants.

By Zhecheng Ren, Xuanji He, Xiaoxiao Li, Zhichen Han, Gaoyang Dong, Gaosheng Zhang, Minchuan Chen, Fengjie Zhu
arXiv Computation and Language
Sep 28

Asymmetric Classifier-Free Guidance for Target-Speaker ASR

The paper introduces asymmetric classifier‑free guidance (CFG) for target‑speaker ASR using Whisper, where a speaker‑conditioned branch predicts the target transcript and a speaker‑unconditioned branch predicts serialized multi‑speaker transcripts. CFG modulates the influence of speaker conditioning during decoding via a single guidance scale, which is first set globally on development data and then refined per utterance by a lightweight encoder‑based predictor while keeping the recognition model fixed. The resulting system yields up to 21.8% relative WER reduction over a condition‑only baseline and 5.6% over standard conditional decoding under domain shifts.

By Yiwen Guan, Jacob Whitehill
Hugging Face Trending Papers
Jun 2

Efficient ASR Training with Conversations that Never Happened

Conversational ASR for lower-resource languages and niche domains is limited by the scarcity of domain-matched multi-speaker training data. We propose an augmentation pipeline that generates scenario-level dialogues with participant metadata, maps speaker attributes to TTS voice profiles, and assembles synthesized utterances into speaker-aware simulated conversations.

arXiv AI
Sep 10

X2Streaming-ASR: wait when uncertain, emit when ready for streaming ASR

X2Streaming-ASR introduces a method for streaming automatic speech recognition that separates the decision of when to commit a transcript from what to commit. The approach uses a three‑stage training process: first establishing streaming capability, then warm‑starting a commit policy with automatically probed trajectories, and finally refining the policy with character‑level, segment‑assigned group‑relative rewards for accuracy and latency. On AISHELL‑1/2/3 and WenetSpeech datasets, the system achieves mean character‑level commit latencies of 27–84 ms, far lower than baseline systems, while also attaining the best streaming character error rates on AISHELL‑1 and AISHELL‑3.

By Zhiwei Lin, Kaiqi Fu, Rime Wen, Zehan Liu, Shawn Qin, Roy Gan, Hao Wang, Qian Wang
arXiv Computation and Language
Sep 15

Enabling Streaming User Transcription in Full-Duplex Speech-to-Speech Models

The paper introduces a lightweight ASR head that can be added to full‑duplex speech‑to‑speech models, enabling real‑time user transcription without major architectural changes. The method adds only a few parameters and preserves full‑duplex conversational features such as turn‑taking and barge‑in. Experiments show a streaming WER of 10.21% within the duplex framework and 7.73% when trained as a standalone ASR model, matching state‑of‑the‑art performance.

By Ke Hu, Nourchene Ferchichi, Edresson Casanova, Ankita Pasad, Elena Rastorgueva, Chen Chen, Nithin Rao Koluguri, Piotr Zelasko, Yifan Peng, Hainan Xu, Zhehuai Chen, Boris Ginsburg
arXiv Computation and Language
Nov 9

Benchmarking Speaker Embedders in the Sub-Second Regime for Real-Time Voice AI Agents

The paper evaluates several open‑source speaker embedding models—TitaNet‑Large, TitaNet‑Small, ECAPA‑TDNN, CAM++, and SpeakerNet—under a strict sub‑second latency constraint for real‑time voice AI agents. Using a controlled benchmark that injects a second speaker into real call recordings and a weighted human‑annotated production set, the authors find that (1) denoising before embedding generally worsens performance, (2) truncating audio below the full voice‑activity segment degrades all models, and (3) about half of the TitaNet embedding dimensions are redundant, encoding demographic attributes that add little discriminative value. They recommend running verification on raw, full voice‑activity audio, setting thresholds to favor precision, and using truncated embeddings to reduce similarity‑search cost.

By Abhishek Gautam
arXiv AI
Jul 15

An Omnilingual-ASR-Based Speech-LLM System for the 2nd MLC-SLM Challenge

arXiv:2607. 12468v1 Announce Type: cross Abstract: We describe our submission to Task 1 of the 2nd MLCSLM Challenge: a cascaded diarization-then-recognition system that combines DiariZen-Large-s80 (WavLM-Large) segmentation, CAM++ embedding-based two-speaker clustering, and a LoRA-adapted omniASR LLM 7B v2 recognizer, with no oracle segmentation or speaker labels at test time.

By Shuming Fang, Shuifei Zeng
arXiv Computation and Language
Oct 28

AzeroS: Extending LLM to Speech with Self-Generated Instruction-Free Tuning

The paper introduces AZeroS, a speech‑adapted large language model that uses Self‑Generated Instruction‑Free Tuning (SIFT) to avoid task‑specific instruction data. By training only two lightweight projection modules on 25,000 hours of speech‑text pairs and 3,000 hours of paralinguistic labels, AZeroS achieves state‑of‑the‑art results on semantic and paralinguistic benchmarks while keeping the core LLM and audio encoder frozen.

By Yiwen Shao, Wei Liu, Jiahong Li, Tianzi Wang, Kun Wei, Meng Yu, Dong Yu
arXiv Computation and Language
Oct 12

Rethinking Entropy Allocation in LLM-based ASR: Understanding the Dynamics between Speech Encoders and LLMs

The paper revisits large‑language‑model (LLM) based automatic speech recognition (ASR) from an entropy‑allocation perspective, introducing three metrics to analyze how entropy reduction is distributed between the speech encoder and the LLM. It proposes a multi‑stage training strategy that addresses entropy‑allocation inefficiencies, including a redesigned pretraining phase to reduce the speech‑text modality gap and an iterative asynchronous fine‑tuning stage to maintain functional decoupling and limit encoder drift. Experiments on Mandarin and English benchmarks demonstrate competitive performance with state‑of‑the‑art models using only 2.3 B parameters, while also reducing hallucinations through the decoupling‑oriented design.

By Yuan Xie, Jiaqi Song, Guang Qiu, Xianliang Wang, Ming Lei, Jie Gao, Jie Wu