← Back to all news
arXiv AI By Yuxiang Zhao, Yichi Zhang, Yanjie An, Yanqiao Zhu, Zhanxun Liu, Yushen Chen, Qixi Zheng, Haina Zhu, Yunchong Xiao, Keqi Deng, Shuai Fan, Kai Yu, Xie Chen

X-Translator: A Real-Time Multilingual Speaker-Aware Speech-to-Speech Translation System

Read the original on arXiv AI →

X-Translator is a low‑cost, modular real‑time speech‑to‑speech translation system that integrates streaming ASR, machine translation, and prompt‑conditioned TTS, managed by a session‑level runtime controller. It uses incremental segment commitment to stabilize ASR streams and an online speaker prompt manager to maintain speaker consistency across multi‑speaker conversations. The system is evaluated on translation quality, speech naturalness, latency, and speaker preservation using OpenSTBench, and its code and demo are publicly available on GitHub.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Computation and Language
Sep 1

Simulstream: Open-Source Toolkit for Evaluation and Demonstration of Streaming Speech-to-Text Translation Systems

Simulstream is an open‑source toolkit designed to evaluate and demonstrate streaming speech‑to‑text translation systems. It supports both incremental and re‑translation decoding on long‑form speech, offers fine‑grained logging for quality and latency metrics, and includes an interactive web interface for real‑time visualization and comparison. The toolkit addresses the fragmented evaluation landscape by providing a unified framework that accommodates different decoding strategies and input formats.

By Marco Gaido, Sara Papi, Mauro Cettolo, Matteo Negri, Luisa Bentivogli
benchmarks
More like this →
arXiv AI
3d ago

X2Streaming-ASR: wait when uncertain, emit when ready for streaming ASR

X2Streaming-ASR introduces a method for streaming automatic speech recognition that separates the decision of when to commit a transcript from what to commit. The approach uses a three‑stage training process: first establishing streaming capability, then warm‑starting a commit policy with automatically probed trajectories, and finally refining the policy with character‑level, segment‑assigned group‑relative rewards for accuracy and latency. On AISHELL‑1/2/3 and WenetSpeech datasets, the system achieves mean character‑level commit latencies of 27–84 ms, far lower than baseline systems, while also attaining the best streaming character error rates on AISHELL‑1 and AISHELL‑3.

By Zhiwei Lin, Kaiqi Fu, Rime Wen, Zehan Liu, Shawn Qin, Roy Gan, Hao Wang, Qian Wang
agentsmultimodal
More like this →
arXiv Computation and Language
2d ago

Streaming Translation and Transcription Through Speech-to-Text Causal Alignment

The paper introduces Hikari, a policy‑free, end‑to‑end model that performs simultaneous speech‑to‑text translation and streaming transcription. It employs a Decoder Time Dilation mechanism to mitigate overuse of WAIT tokens during training and a supervised fine‑tuning strategy that helps the model recover from delays, improving the quality‑latency trade‑off. Despite its modest size, Hikari achieves competitive translation quality at consistently low latency, outperforming larger published IWSLT 2026 submissions and proprietary API systems on en‑ja, en‑de, and en‑ru tasks.

By Roman Koshkin, Jeon Haesung, Lianbo Liu, Hao Shi, Mengjie Zhao, Yusuke Fujita, Yui Sudo
nlpfine-tuningsafety
More like this →