arXiv Computation and Language

EmphTTS: an emphasis-control TTS with reinforcement learning

EmphTTS is a non‑autoregressive text‑to‑speech system that uses Group Relative Policy Optimization (GRPO) to train its duration predictor with an emphasis localization reward, allowing direct word‑level emphasis control. The system outperforms existing baselines in both objective emphasis evaluation and subjective preference tests, and ablation studies confirm that GRPO improves emphasis realization while reducing mismatch between the duration predictor and the TTS model.

arXiv Computation and Language

5d ago

Beyond Speech Captions: Speech-Rewarded Style Planning for Conversational Text-to-Speech

The paper introduces Speech-Rewarded Style Planning (SRSP), a method that trains a text-based style planner using a frozen downstream TTS model. SRSP generates candidate style instructions from dialogue history and response text, optimizing them with group-relative policy optimization (GRPO) where the reward is the teacher‑forced likelihood of target speech tokens. Experiments on an English subset of the ISCSLP 2026 CoT‑TTS corpus show that SRSP outperforms baseline LLM and target‑audio‑informed captioning approaches in speech‑style and emotion similarity, mel‑cepstral distortion, and contextual appropriateness. "whyItMatters":"The study demonstrates a more effective way to align natural‑language style descriptions with actual acoustic output, improving controllability and fidelity in conversational text‑to‑speech systems."

By Shiao Zhu, Lianbo Liu, Sizhen Lyu, Yuzhe Wang, Sheng Li, Takahiro Shinozaki

llms multimodal safety

arXiv Computation and Language

Sep 1

Sequential Trajectories and Simultaneous Blending: Multi-Emotion Modeling for Instruction-Following TTS

arXiv:2608.30325v1 Announce Type: new Abstract: Natural-language instructions enable flexible control of synthesized speech, yet emotional TTS systems primarily model a single utterance-level affect,...

By Yan Zhou, Yun Hong, Yang Feng

fine-tuning multimodal

arXiv Computation and Language

Sep 15

Reducing the Output-Mode Gap in Speech Language Models via Joint-Output On-Policy Distillation

arXiv:2609.15313v1 Announce Type: cross Abstract: Autoregressive generation of interleaved text and acoustic tokens is a common approach to spoken-response generation in speech large language models....

By Daxin Tan, Dehua Tao, Chengxi Deng, Hanlin Zhang, Xiao Chen

llms fine-tuning efficiency multimodal

arXiv Computation and Language

Sep 1

When Does Predictor-Based RL Align with Human Perception? A Study of Subjective Rewards in Codec-Based Speech Language Models

arXiv:2608.31035v1 Announce Type: new Abstract: Codec-based text-to-speech (TTS) models make language-model post-training applicable to speech generation, but it remains unclear when learned perceptu...

By Joonyong Park, Jerry Li

llms rag reinforcement-learning multimodal safety

arXiv AI

Jun 26

VoiceTTA: Enhancing Zero-Shot Text-to-Speech via Reinforcement Learning-Based Test-Time Adaptation

arXiv:2606. 26534v1 Announce Type: cross Abstract: Recently, zero-shot text-to-speech (TTS) has enabled high-fidelity and expressive speech synthesis, but it often fails to imitate unseen speaking styles from uncommon scenarios (e.

By Tianxin Xie, Chenxing Li, Dong Yu, Li Liu

diffusion reinforcement-learning fine-tuning multimodal benchmarks

arXiv Computation and Language

Sep 22

COT-TTS: Audio Context-Aware Text-to-Speech with Chain-of-Thought Reasoning

arXiv:2609.22697v1 Announce Type: new Abstract: Recently, text-to-speech systems have made significant progress in speech expressiveness and controllability. However, the speaking style of generated...

By Weizhen Bian, Sitong Cheng, Rongxiu Zhong, Jiahao Pan, Liumeng Xue, Boyi Kang, Shilei Zhang, Jinglei Liu, Yue Wang, Junlan Feng, Bei Liu, Wei Xue

multimodal benchmarks

arXiv AI

Sep 4

HOMURA: Taming the Sand-Glass for Time-Constrained LLM Translation via Reinforcement Learning

The paper introduces Sand-Glass, a benchmark for evaluating translation under syllable-level duration constraints, and proposes Homura, a reinforcement learning framework that optimizes the trade-off between semantic preservation and temporal compliance. Homura uses a constrained reinforcement learning objective with a dynamic syllable-ratio reward to effectively control output length. Experimental results show that Homura outperforms strong baselines, achieving precise length control while maintaining semantic adequacy.

By Ziang Cui, Mengran Yu, Chenyu Shi, Yingxuan Shi, Tianjiao Li

llms reinforcement-learning benchmarks safety

arXiv Computation and Language

Sep 1

UltraVoice: Scaling Fine-Grained Style-Controlled Speech Conversations for Spoken Dialogue Models

arXiv:2510.22588v2 Announce Type: replace-cross Abstract: Spoken dialogue models currently lack the ability for fine-grained speech style control, a critical capability for human-like interaction tha...

By Wenming Tu, Guanrou Yang, Ruiqi Yan, Wenxi Chen, Ziyang Ma, Yipeng Kang, Kai Yu, Xie Chen, Zilong Zheng

nlp fine-tuning multimodal benchmarks

arXiv Computation and Language

Sep 23

Transcribe, Translate, and Optimize: Joint Reward Learning for Speech Translation

arXiv:2609.26536v1 Announce Type: new Abstract: In LLM-based speech translation, transcription-based chain-of-thought (CoT) suffers from a mismatch between reference transcripts used in supervised fi...

By Yanghe Dong, Wanting Huang, Weiran Wang

llms fine-tuning

arXiv AI

Aug 5

GROW: Group-Relative Advantage-Weighted On-Policy Reinforcement Learning of Autoregressive-Diffusion Text-to-Speech model

arXiv:2608. 03215v1 Announce Type: cross Abstract: Reinforcement learning for flow-matching text-to-speech is complicated by deterministic ODE sampling: trajectory-level policy-gradient methods typically convert the ODE into an SDE and track per-step likelihood ratios, introducing stochastic perturbations and substantial overhead.

By Guanrou Yang, Tian Tan, Qian Chen, Ziyang Ma, Yakun Song, Zhikang Niu, Qi Chen, Wenming Tu, Haitao Li, Shan Yang, Xie Chen

diffusion reinforcement-learning multimodal

arXiv Computation and Language

Sep 30

HPRO: Hierarchical Progressive Reward Optimization via Preference Extraction for Emotional Text-to-Speech

arXiv:2606.28249v2 Announce Type: replace-cross Abstract: Recently, Large Language Model (LLM)-based Text-to-Speech (TTS) models have achieved remarkable naturalness. However, the standard Supervised...

By Sihang Nie, Xiaofen Xing, Rui Xing, Haoming Li, Ruitong Xiao, Jingyuan Xing, Baiji Liu, Xiangmin Xu

llms diffusion reinforcement-learning fine-tuning multimodal

arXiv AI

Sep 21

Listen Before You Speak: Response Planning from Listener Facial Reactions for Conversational Speech Generation

The paper introduces ReACT‑TTS, a two‑stage framework that uses a one‑second sequence of listener facial reactions to plan the emotion and prosody of a conversational utterance before speech synthesis. Experiments on the MELD dataset show that temporally conditioned models outperform text‑only baselines in macro‑F1 and VAD concordance, while accuracy remains unchanged. A contextual‑appropriateness study with speech researchers indicates a strong preference for temporally conditioned responses, and the approach is integrated with a Grad‑TTS backbone for end‑to‑end speech generation.

By Yunji Chu

Related stories