arXiv Computation and Language
Nov 5

CRESiST: Self-Enhancing Explorative Policy Learning for Simultaneous Speech Translation

CRESiST introduces an end‑to‑end interleaved read‑write framework that leverages an omni‑modal LLM for simultaneous speech translation. By interleaving speech chunks with text tokens, the system fully reuses the KV cache and supports segmentation‑free, infinite‑length inference, eliminating the need for external voice activity detection. An Actor‑Critic self‑enhancement pipeline trains an RNN‑T actor to sample read‑write trajectories, while an LLM‑based critic evaluates them using semantic and latency rewards, distilling the optimal streaming policy back to the actor. Experiments on sentence‑ and document‑level benchmarks show that CRESiST sets new state‑of‑the‑art quality‑latency Pareto frontiers, outperforming existing cascaded and static‑alignment baselines.

By Chenyang Le, Pei Zhang, Yuxiang Chen, Bing Han, Derek F. Wong, Baosong Yang, Yanmin Qian
arXiv Computation and Language
Oct 5

Learning When to Commit from Partial Speech for End-to-End Simultaneous Speech Translation

The paper presents a method for improving simultaneous speech translation by adapting a full‑utterance speech language model with prefix supervision derived from its own complete and partial waveform translations, eliminating the need for transcripts or human translations. Experiments on FLEURS and CoVoST2 across three language directions show that prefix training enhances quality–latency trade‑offs, especially when combined with multi‑turn append‑only decoding, and that a confidence threshold effectively controls the inference‑time quality–latency balance. The study also explores the impact of synthesis margin on translation quality and calibration, finding a non‑monotonic relationship with latency.

By Hieu Hoang, Amittai Axelrod
arXiv Computation and Language
Sep 28

All In Good Time: Causality-Aware Framework for LLM-Based Simultaneous Speech-to-Speech Translation

The paper introduces FAST-CAP, a causality‑aware framework for simultaneous speech‑to‑speech translation that combines a factorized S2ST architecture, an adaptive policy, and a new latency metric. It employs a novel data pipeline to generate high‑fidelity, causally aligned segments, improving voice transfer and reducing the need for large training datasets. Experiments on Spanish, German, and French demonstrate that FAST‑CAP outperforms fixed‑policy baselines, achieving up to +1.2 BLEU, 26% lower latency, and a 38.8% relative latency reduction while maintaining speaker fidelity.

By Amir Hussein, Enas Albasiri, Travis M. Bartley, Nourchene Ferchichi, Ke Hu, Harishchandra Dubey, Myungjong Kim, Zhehuai Chen, Oluwatobi Olabiyi, Sanjeev Khudanpur