← Back to all news
arXiv Machine Learning By Yuanzhong Zheng, Wei Huang, Ming Ma, Kailun Jin, Youhai Jiang

Decodability Is Not Controllability: Local and Anisotropic Speaker Control in Continuous-Latent TTS

Read the original on arXiv Machine Learning →

The paper demonstrates that high linear decodability in continuous‑latent TTS does not guarantee causal controllability of speaker identity. In VoxCPM2, probes at the final residual acoustic language model layer show strong accuracy, yet aligned edits fail to produce target‑dominant dyads, whereas early‑position edits succeed. Cross‑architecture experiments reveal that backbone edits shift the source voice without attracting the target, while x‑vector ablation eliminates source similarity, underscoring the need for geometric specificity and a computational path that embeds the intended speaker condition into synthesis.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Sep 4

VoxReason: Listener-Free Evaluation of Source-Grounded Speech Planning Before Synthesis

VoxReason introduces a listener‑free evaluation framework that measures whether a speech planning system’s delivery choices—such as pitch, energy, rate, pause, emphasis, and stance—are grounded in cited source records before any waveform is generated. The system outputs a source‑cited speaking plan and uses a deterministic verifier to check citation legality, slot agreement, unsupported states, schema validity, and counterfactual locality. Experiments on 1,440 source‑label cases show that simple slot accuracy can be misleading, while a 7B locality‑based repair model significantly improves plan‑slot accuracy and locality, and removing source records sharply reduces the grounded score. whyItMatters":"The framework provides a concrete, measurable way to ensure that expressive speech systems make source‑licensed planning decisions, addressing a source‑use failure that occurs before audio synthesis."

By Mengzhe Geng
agentsfine-tuning
More like this →
arXiv AI
Jul 24

AG-REPA: Causal Layer Selection for Representation Alignment in Audio Flow Matching

arXiv:2603. 01006v3 Announce Type: replace-cross Abstract: REPresentation Alignment (REPA) improves the training of generative flow models by aligning intermediate hidden states with pretrained teacher features, but its effectiveness in token-conditioned audio Flow Matching critically depends on the choice of supervised layers, which is typically made heuristically based on the depth.

By Pengfei Zhang, Tianxin Xie, Minghao Yang, Li Liu
diffusionbenchmarkssafety
More like this →
arXiv Machine Learning
Oct 21

Steering intonational tunes in text-to-speech synthesis

The study investigates how intonation is encoded in text‑to‑speech models that use residual vector quantization. By probing different components of the architecture, the authors find that the transformer backbone’s hidden state carries the nuclear intonation pattern, while the first token level does not. Steering this backbone state with vectors derived from human‑imitated intonation patterns can shift the generated speech toward falling or rising melodies, though precise control remains limited and naturalness slightly decreases.

By Juan Pablo Vigneaux, Jennifer Cole
llmsefficiencymultimodal
More like this →