The paper introduces FAST-CAP, a causality‑aware framework for simultaneous speech‑to‑speech translation that combines a factorized S2ST architecture, an adaptive policy, and a new latency metric. It employs a novel data pipeline to generate high‑fidelity, causally aligned segments, improving voice transfer and reducing the need for large training datasets. Experiments on Spanish, German, and French demonstrate that FAST‑CAP outperforms fixed‑policy baselines, achieving up to +1.2 BLEU, 26% lower latency, and a 38.8% relative latency reduction while maintaining speaker fidelity.
By Amir Hussein, Enas Albasiri, Travis M. Bartley, Nourchene Ferchichi, Ke Hu, Harishchandra Dubey, Myungjong Kim, Zhehuai Chen, Oluwatobi Olabiyi, Sanjeev Khudanpur
DIAL‑S2ST is a post‑training method that learns to control inference‑time latency in simultaneous speech‑to‑speech translation models. It reuses pretrained translation capabilities through self‑distillation from lookahead predictions and limits large text distillation losses to address the imitation gap, providing supervision for earlier emission without paired target data or reward computation. By jointly training a single LoRA adapter across scales with matched lookahead guidance, the learned timing can be adjusted at inference, and experiments on French‑to‑English translation with Hibiki show that only 10 hours of source speech enable this adaptation, yielding flexible latency control with a favorable quality‑latency trade‑off.
By Jeongsoo Choi, Junseok Ahn, Joon Son Chung
The paper introduces two new policies—Recent Frame Attention Policy (RFAP) and Dual-Condition Attention Policy (DCAP)—for simultaneous speech-to-text translation. These policies leverage the cross‑attention mechanism of encoder‑decoder models to determine optimal moments for partial translation, enabling offline models to operate in streaming scenarios without extra training. Experiments on the CVSS‑C corpus show RFAP improves BLEU scores by up to 4.0 points while cutting delay by nearly one second, and DCAP maintains high quality at very low latency.
By Filip T\u{a}\c{s}\u{a}dan, Ema Tomanov\'a, Ondrej Lopuch, Pawe{\l} Bilko, Anders S{\o}gaard
The paper introduces a training strategy for cascaded simultaneous speech translation that allows the system to dynamically decide how much of the source prefix to translate. By fine‑tuning a large language model (Qwen3‑8B) on stable prefixes—pairs of source prefixes and the longest shared translation with the full sentence—the authors enable contextual read‑write decisions beyond fixed wait‑k or target‑suffix deletion. Experiments on English‑to‑German, Japanese, and Chinese demonstrate that stable prefixes improve the quality‑latency tradeoff across various test sets.
By Hieu Hoang, Amittai Axelrod, Matt Post
Uni-Stream is a unified streaming speech language model that supports both speech recognition and synthesis within a single architecture. Built on compact pretrained speech models, it processes interleaved text and speech in real time and achieves performance comparable to task‑specific models through a two‑stage training approach. The authors also introduce a data annotation pipeline for streaming training and plan to release the generated transcriptions and word‑level timestamps, while analyzing how speech and text representations occupy distinct latent spaces in the model.
By Xuan Shi, Chang Zeng, Tiantian Feng, Ye Zhang, Jun Liu, Shih-Heng Wang, Lie Lu, Shrikanth Narayanan
arXiv:2504.11809v2 Announce Type: replace
Abstract: Simultaneous speech translation (SimulST) produces translations incrementally while processing partial speech input. Although large language models...
By Biao Fu, Donglei Yu, Minpeng Liao, Chengxi Li, Xinjie Chen, Yidong Chen, Kai Fan, Xiaodong Shi