← Back to all news
arXiv Machine Learning

FASTDIAR: Frame-level speaker encoder for Streaming Diarization

FASTDIAR is a frame‑level speaker encoder designed for real‑time streaming diarization on a single CPU thread. It converts a state‑of‑the‑art speaker recognition architecture into a causal model that emits an embedding every 80 ms from a two‑second window, and pairs it with online clustering that updates only when the stream’s self‑similarity changes. Trained via distillation from an utterance‑level teacher, it achieves the highest accuracy on low‑overlap benchmarks, degrades less with more speakers, and runs five times faster than real time.

Related stories

arXiv AI
Sep 17

Look Less, Hear Better: Jointly Rewarded GRPO for Streaming ASR

The paper introduces AWED, a word‑level emission‑delay metric, and demonstrates that post‑training a delayed‑streaming model with a joint reward (GRPO) improves both accuracy and latency. Using a single operating point (τ=6 frames), the method outperforms both its supervised baseline and the Voxtral Realtime backbone across all lookahead budgets, reducing WER by up to 30.8% at 80 ms delay and lowering median AWED from 1.17 s to 1.04 s.

By Xiuwen Zheng
llmsfine-tuningmultimodalsafety
More like this →
arXiv AI
Sep 10

X2Streaming-ASR: wait when uncertain, emit when ready for streaming ASR

X2Streaming-ASR introduces a method for streaming automatic speech recognition that separates the decision of when to commit a transcript from what to commit. The approach uses a three‑stage training process: first establishing streaming capability, then warm‑starting a commit policy with automatically probed trajectories, and finally refining the policy with character‑level, segment‑assigned group‑relative rewards for accuracy and latency. On AISHELL‑1/2/3 and WenetSpeech datasets, the system achieves mean character‑level commit latencies of 27–84 ms, far lower than baseline systems, while also attaining the best streaming character error rates on AISHELL‑1 and AISHELL‑3.

By Zhiwei Lin, Kaiqi Fu, Rime Wen, Zehan Liu, Shawn Qin, Roy Gan, Hao Wang, Qian Wang
agentsmultimodal
More like this →
arXiv Computation and Language
Sep 28

Inference-Time Target Speaker Unlearning in LLM-Based Automatic Speech Recognition

The paper introduces a new target‑speaker unlearning task for automatic speech recognition (TSU‑ASR) that allows certain speakers to opt out of transcription while still indicating their presence. A lightweight Enrollment‑Conditioned Gating (ECG) module is added to a frozen dual‑stream speech LLM, enabling dynamic unlearning of new opt‑out speakers during inference. Experiments on AMI and AliMeeting datasets show significant drops in transcription accuracy for opt‑out speakers while preserving performance for retained speakers.

By Bo Su, Yueru Yan, Thai Le
llmsmultimodalsafety
More like this →
arXiv Machine Learning
Sep 22

AVTR-1: Open Stack for Real-Time Interactive Avatars

arXiv:2609.22913v1 Announce Type: cross Abstract: Talking-head and dyadic models now achieve real-time inference, yet fast motion generation alone does not produce an interactive conversation. A live...

By Artem Kravtsov, Dmitrii Ziganshin, Vsevolod Poletaev, Gleb Balitskiy, Anastasia Tikhonova, Egor Burkov, Vadim Lebedev
agentsefficiency
More like this →
arXiv Machine Learning
Sep 7

GEPARD - Generative, Prosody-aware, Autoregressive text-to-speech model for Realtime Dialogue

GEPARD is a streaming text‑to‑speech model that uses a standard large language model backbone to generate speech autoregressively, decoding audio with an FSQ‑based neural codec. It streams audio chunk‑by‑chunk as text arrives, achieving a real‑time factor of about 0.067 and an aggregate speedup of roughly 204× on a single GPU with 256 concurrent streams. The design keeps all complex auxiliary mechanisms outside the decode loop, enabling deployment with a standard LLM engine (vLLM) without kernel modifications.

By Denis Pavlov, Ulanbek Abdurazakov, Nursultan Bakashov
llmsragreinforcement-learningefficiencymultimodal
More like this →
arXiv AI
Jul 7

StarTSE: Towards Streaming Target Speaker Extraction via Chunk-wise Interleaved Splicing of Autoregressive Language Model

arXiv:2604. 19635v2 Announce Type: replace-cross Abstract: While generative models have set new benchmarks for Target Speaker Extraction (TSE), their inherent reliance on global context precludes deployment in real-time applications.

By Shuhai Peng, Hui Lu, Jinjiang Liu, Liyang Chen, Guiping Zhong, Jiakui Li, Huimeng Wang, Haiyun Li, Liang Cao, Shiyin Kang, Zhiyong Wu
llmsbenchmarks
More like this →