CRESiST introduces an end‑to‑end interleaved read‑write framework that leverages an omni‑modal LLM for simultaneous speech translation. By interleaving speech chunks with text tokens, the system fully reuses the KV cache and supports segmentation‑free, infinite‑length inference, eliminating the need for external voice activity detection. An Actor‑Critic self‑enhancement pipeline trains an RNN‑T actor to sample read‑write trajectories, while an LLM‑based critic evaluates them using semantic and latency rewards, distilling the optimal streaming policy back to the actor. Experiments on sentence‑ and document‑level benchmarks show that CRESiST sets new state‑of‑the‑art quality‑latency Pareto frontiers, outperforming existing cascaded and static‑alignment baselines.
By Chenyang Le, Pei Zhang, Yuxiang Chen, Bing Han, Derek F. Wong, Baosong Yang, Yanmin Qian
The paper introduces Hikari, a policy‑free, end‑to‑end model that performs simultaneous speech‑to‑text translation and streaming transcription. It employs a Decoder Time Dilation mechanism to mitigate overuse of WAIT tokens during training and a supervised fine‑tuning strategy that helps the model recover from delays, improving the quality‑latency trade‑off. Despite its modest size, Hikari achieves competitive translation quality at consistently low latency, outperforming larger published IWSLT 2026 submissions and proprietary API systems on en‑ja, en‑de, and en‑ru tasks.
By Roman Koshkin, Jeon Haesung, Lianbo Liu, Hao Shi, Mengjie Zhao, Yusuke Fujita, Yui Sudo
The paper presents a method for improving simultaneous speech translation by adapting a full‑utterance speech language model with prefix supervision derived from its own complete and partial waveform translations, eliminating the need for transcripts or human translations. Experiments on FLEURS and CoVoST2 across three language directions show that prefix training enhances quality–latency trade‑offs, especially when combined with multi‑turn append‑only decoding, and that a confidence threshold effectively controls the inference‑time quality–latency balance. The study also explores the impact of synthesis margin on translation quality and calibration, finding a non‑monotonic relationship with latency.
By Hieu Hoang, Amittai Axelrod
arXiv:2504.11809v2 Announce Type: replace
Abstract: Simultaneous speech translation (SimulST) produces translations incrementally while processing partial speech input. Although large language models...
By Biao Fu, Donglei Yu, Minpeng Liao, Chengxi Li, Xinjie Chen, Yidong Chen, Kai Fan, Xiaodong Shi
arXiv:2610.35042v1 Announce Type: new
Abstract: Simultaneous speech translation (SimulST) aims to translate ongoing speech with low latency while preserving translation quality. Recent studies have a...
By Haotian Tan, Hiroki Ouchi, Sakriani Sakti
Simultaneous speech translation (SimulST) aims to translate ongoing speech with low latency while preserving translation quality. Recent studies have adopted large language models (LLMs) for SimulST a...