The paper introduces ReACT‑TTS, a two‑stage framework that uses a one‑second sequence of listener facial reactions to plan the emotion and prosody of a conversational utterance before speech synthesis. Experiments on the MELD dataset show that temporally conditioned models outperform text‑only baselines in macro‑F1 and VAD concordance, while accuracy remains unchanged. A contextual‑appropriateness study with speech researchers indicates a strong preference for temporally conditioned responses, and the approach is integrated with a Grad‑TTS backbone for end‑to‑end speech generation.
arXiv:2609.40317v1 Announce Type: new Abstract: While talking head generation has advanced rapidly, generating natural listener behavior in dyadic conversations, which know when to react, how to reac...
arXiv:2608.16053v2 Announce Type: replace Abstract: Synthetic conversational speech has become an important resource for developing and evaluating conversational speech systems. However, existing dia...
The paper introduces Speech-Rewarded Style Planning (SRSP), a method that trains a text-based style planner using a frozen downstream TTS model. SRSP generates candidate style instructions from dialogue history and response text, optimizing them with group-relative policy optimization (GRPO) where the reward is the teacher‑forced likelihood of target speech tokens. Experiments on an English subset of the ISCSLP 2026 CoT‑TTS corpus show that SRSP outperforms baseline LLM and target‑audio‑informed captioning approaches in speech‑style and emotion similarity, mel‑cepstral distortion, and contextual appropriateness. "whyItMatters":"The study demonstrates a more effective way to align natural‑language style descriptions with actual acoustic output, improving controllability and fidelity in conversational text‑to‑speech systems."
arXiv:2609.16396v1 Announce Type: new Abstract: Negation is typically modeled through its linguistic realization, although spoken interaction is accompanied by tightly coordinated nonverbal behavior....
arXiv:2609.22697v1 Announce Type: new Abstract: Recently, text-to-speech systems have made significant progress in speech expressiveness and controllability. However, the speaking style of generated...
The paper introduces a method called randomized intermediate guidance for training tandem speech-to-speech models, where a large language model (LLM) acts as a backend providing candidate responses while the user is speaking. Instead of simulating the backend’s guidance, the approach derives guidance directly from the conversation corpus, using target responses for informative guidance and randomly sampled responses to simulate irrelevant updates. Experiments on synthetic dialogues and 3.8k hours of real conversations show that this technique yields response quality comparable to LLM-generated baselines while improving natural turn‑taking and audio‑judge naturalness.
arXiv:2608. 19515v1 Announce Type: new Abstract: Prosodic cues can convey task-relevant information that alters the trajectory and outcome of a task-oriented dialogue, even when the words themselves remain unchanged.
Prosodic cues can convey task-relevant information that alters the trajectory and outcome of a task-oriented dialogue, even when the words themselves remain unchanged. Yet existing benchmarks typically evaluate prosodic perception, response appropriateness, and task-oriented dialogue in isolation, making it difficult to test whether prosodic evidence changes downstream decisions.
Streaming speech-to-speech language models aim to answer spoken queries directly with synthetic speech. However, standard speech and text benchmarks do not capture whether these systems behave naturally in conversations, where timing, turn-taking, prosody, interpersonal stance, language and dialect consistency, and relationship-aware appropriateness jointly shape perceived quality.
The paper presents Conversational Voice Aesthetic Model (CVAM), a speech large language model that describes voice aesthetics in conversational contexts. CVAM predicts nine categorical attributes such as gender, pitch, pacing, emotion, and delivery, and is trained on human-annotated data from the CANDOR corpus. Experiments show that CVAM aligns better with human listeners than Gemini 3.1 Pro and other open‑source speech LLMs, demonstrating the value of grounding voice aesthetics in human perception.
arXiv:2608.30325v1 Announce Type: new Abstract: Natural-language instructions enable flexible control of synthesized speech, yet emotional TTS systems primarily model a single utterance-level affect,...
EmoTra‑TTS introduces a method for smooth intra‑utterance emotion transitions in speech synthesis. It uses a multi‑pass flow blending pipeline, dual‑stage VAD conditioning, and direction‑magnitude decoupled injection to generate frame‑aligned emotional prosody. The system adds only 0.43% more parameters, incurs no latency, and outperforms four state‑of‑the‑art baselines and two commercial systems in emotion transition quality and overall preference tests.