Efficient and Adaptive Simultaneous Speech Translation with Fully Unidirectional Architecture
Read the original on arXiv Computation and Language →The Flow has not summarised this story yet — read it at arXiv Computation and Language.
The Flow has not summarised this story yet — read it at arXiv Computation and Language.
CRESiST introduces an end‑to‑end interleaved read‑write framework that leverages an omni‑modal LLM for simultaneous speech translation. By interleaving speech chunks with text tokens, the system fully reuses the KV cache and supports segmentation‑free, infinite‑length inference, eliminating the need for external voice activity detection. An Actor‑Critic self‑enhancement pipeline trains an RNN‑T actor to sample read‑write trajectories, while an LLM‑based critic evaluates them using semantic and latency rewards, distilling the optimal streaming policy back to the actor. Experiments on sentence‑ and document‑level benchmarks show that CRESiST sets new state‑of‑the‑art quality‑latency Pareto frontiers, outperforming existing cascaded and static‑alignment baselines.
arXiv:2610.35042v1 Announce Type: new Abstract: Simultaneous speech translation (SimulST) aims to translate ongoing speech with low latency while preserving translation quality. Recent studies have a...
Simultaneous speech translation (SimulST) aims to translate ongoing speech with low latency while preserving translation quality. Recent studies have adopted large language models (LLMs) for SimulST a...
The paper presents a method for improving simultaneous speech translation by adapting a full‑utterance speech language model with prefix supervision derived from its own complete and partial waveform translations, eliminating the need for transcripts or human translations. Experiments on FLEURS and CoVoST2 across three language directions show that prefix training enhances quality–latency trade‑offs, especially when combined with multi‑turn append‑only decoding, and that a confidence threshold effectively controls the inference‑time quality–latency balance. The study also explores the impact of synthesis margin on translation quality and calibration, finding a non‑monotonic relationship with latency.
arXiv:2509.21932v3 Announce Type: replace Abstract: How to make human-interpreter-like read/write decisions for simultaneous speech translation (SimulST) systems? Current state-of-the-art systems for...
The paper introduces FAST-CAP, a causality‑aware framework for simultaneous speech‑to‑speech translation that combines a factorized S2ST architecture, an adaptive policy, and a new latency metric. It employs a novel data pipeline to generate high‑fidelity, causally aligned segments, improving voice transfer and reducing the need for large training datasets. Experiments on Spanish, German, and French demonstrate that FAST‑CAP outperforms fixed‑policy baselines, achieving up to +1.2 BLEU, 26% lower latency, and a 38.8% relative latency reduction while maintaining speaker fidelity.