Steering intonational tunes in text-to-speech synthesis

The study investigates how intonation is encoded in text‑to‑speech models that use residual vector quantization. By probing different components of the architecture, the authors find that the transformer backbone’s hidden state carries the nuclear intonation pattern, while the first token level does not. Steering this backbone state with vectors derived from human‑imitated intonation patterns can shift the generated speech toward falling or rising melodies, though precise control remains limited and naturalness slightly decreases.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

• llms
• efficiency
• multimodal