arXiv Computation and Language

Kraken: LLM-based Speech-to-Speech Translation via Low-bitrate VQ and Dual-path Source Conditioning

arXiv AI
Jun 9

End-to-End Training for Discrete Token LLM based TTS System

arXiv:2606. 09234v1 Announce Type: cross Abstract: Recent state-of-the-art (SOTA) text-to-speech (TTS) systems typically adopt a cascaded pipeline consisting of a speech tokenizer, an autoregressive large language model (LLM), and a diffusion based flow-matching (FM) model, with these components trained independently.

By Changfeng Gao, Yong Ren, Jun Yuan, Ye Bai, Zhao You, ShiDong Shang
arXiv Machine Learning
Aug 27

BRIDLE: Generalized Self-supervised Learning with Quantization

BRIDLE is a self‑supervised encoder pretraining framework that extends bidirectional training to audio, image, and video by incorporating residual quantization (RQ) with multiple hierarchical codebooks. This approach allows fine‑grained discretization of latent representations and interleaves training between the encoder and tokenizer. Experiments show that BRIDLE achieves state‑of‑the‑art results on audio classification benchmarks and competitive performance on image and video classification tasks, outperforming traditional vector‑quantization methods.

By Hoang M. Nguyen, Satya N. Shukla, Qiang Zhang, Hanchao Yu, Sreya D. Roy, Dipesh Tamboli, Taipeng Tian, Lingjiong Zhu, Yuchen Liu