arXiv Machine Learning

PitchFlower: A flow-based neural audio codec with pitch controllability

PitchFlower is a flow‑based neural audio codec that offers explicit pitch controllability by flattening and randomly shifting F0 contours during training while conditioning on the true F0 to reconstruct the original audio. A vector‑quantization bottleneck blocks pitch recovery, and a flow‑based decoder produces high‑quality audio. Experiments demonstrate that PitchFlower matches DSP baselines in pitch accuracy, surpasses state‑of‑the‑art neural codecs in audio quality, and remains robust even when trained on WORLD‑transformed audio, effectively removing vocoder artifacts.

Hugging Face Trending Papers

Jun 3

CleanCodec: Efficient and Robust Speech Tokenization via Perceptually Guided Encoding

Neural audio codecs are a key component of speech processing pipelines, compressing audio into discrete tokens for downstream modeling. However, existing codecs struggle to balance reconstruction quality with token efficiency, often encoding perceptually irrelevant information such as background noise and recording artifacts at the expense of linguistically and acoustically meaningful content.

nlp multimodal benchmarks

arXiv AI

Jun 2

DSA-Tokenizer: Disentangled Semantic-Acoustic Tokenization via Flow Matching-based Hierarchical Fusion

arXiv:2601. 09239v5 Announce Type: replace-cross Abstract: Speech tokenizers are a key building block of fully discrete Speech LLMs.

By Hanlin Zhang, Daxin Tan, Dehua Tao, Xiao Chen, Haochen Tan, Yunhe Li, Yuchen Cao, Linqi Song

llms diffusion nlp fine-tuning multimodal

arXiv AI

Jun 9

BareWave: Waveform-Native Flow-Matching Text-to-Speech

arXiv:2606. 09048v1 Announce Type: cross Abstract: Removing intermediate representations and separately trained decoding stages has become an important direction in generative modeling.

By Wei Fan, Chao-Hong Tan, Qian Chen, Wen Wang, Xiangang Li, Kejiang Chen, Weiming Zhang, Nenghai Yu

multimodal safety

arXiv AI

Sep 16

LACE: Layer-Wise Compression for Dynamic Frame Rate Codecs

arXiv:2609.17509v1 Announce Type: cross Abstract: Neural audio codecs are a key component in speech language modeling. However, their high frame rates lead to long sequence lengths, increasing comput...

By Thanapat Trachu, Samuele Cornell, William Chen, Shinji Watanabe

rag computer-vision efficiency multimodal safety

arXiv AI

Jul 2

Enhancing Flow Matching with A Unified Guidance Framework for Efficient and Robust Speech Synthesis

arXiv:2607. 00363v1 Announce Type: cross Abstract: Flow Matching (FM) has emerged as a powerful paradigm for speech generation but remains constrained by high inference latency and timbre leakage.

By Zuda Yu, Qianhui Xu, Ting Chen, Junhui Zhang, Tao Fu, Hongjiang Yu, Qiangqing Wang, Yang Song

diffusion efficiency benchmarks

Hugging Face Trending Papers

Sep 15

LACE: Layer-Wise Compression for Dynamic Frame Rate Codecs

Neural audio codecs are a key component in speech language modeling. However, their high frame rates lead to long sequence lengths, increasing computational costs. Dynamic frame rate codecs mitigate t...

rag computer-vision efficiency multimodal safety

Hugging Face Trending Papers

Sep 21

P2Flow: Phoneme-aware Progressive Flow Matching for Extreme Speech Super-Resolution

Generative models have recently demonstrated considerable promise in speech super-resolution (SSR). Nevertheless, the majority of existing work has concentrated on standard or versatile SSR configurat...

diffusion benchmarks

arXiv Machine Learning

Sep 22

P2Flow: Phoneme-aware Progressive Flow Matching for Extreme Speech Super-Resolution

arXiv:2609.24138v1 Announce Type: cross Abstract: Generative models have recently demonstrated considerable promise in speech super-resolution (SSR). Nevertheless, the majority of existing work has c...

By Ningyuan Yang, Yize Li, Pu Zhao, Diego A. Cuji, Kanad Sarkar, Ryan M. Corey, Xue Lin, Andrew C. Singer

diffusion benchmarks

arXiv Machine Learning

Aug 7

LILAC: An Idempotent Neural Speech Codec

arXiv:2608. 05727v1 Announce Type: cross Abstract: Neural Audio Codecs are widely adopted in speech generation and editing.

By June Young Yi, Dongwook Lee, Jiheum Yeom, Sungroh Yoon

benchmarks

arXiv AI

Sep 7

SCAPES: Semantically Conditioned Autoregressive Prior for Environmental Sounds

SCAPES is a lightweight, resource‑efficient generative model that synthesizes high‑fidelity environmental sounds with high‑level semantic control. It operates on the continuous latent manifold of a neural audio codec, using a segmentation strategy and a Continuous Normalizing Flow to model latent trajectories. A 36‑million‑parameter instance can be trained on limited, uncurated data with a single consumer‑grade GPU, achieving convergence in roughly twice the source audio duration and enabling smooth semantic interpolation.

By Esteban Guti\'errez, Lonce Wyse, Frederic Font, Xavier Serra

diffusion computer-vision nlp

arXiv AI

Aug 5

UniPASE: A Generative Model for Universal Speech Enhancement with High Fidelity and Low Hallucinations

arXiv:2604. 14606v2 Announce Type: cross Abstract: Universal speech enhancement (USE) aims to restore speech signals from diverse distortions across multiple sampling rates.

By Xiaobin Rong, Zheng Wang, Yushi Wang, Jun Gao, Jing Lu

fine-tuning efficiency benchmarks safety

arXiv AI

Jul 1

BEST-RQ-2: Contextualize-Then-Predict, a Two-Step Approach for Self-Supervised Audio Representations

arXiv:2606. 30700v1 Announce Type: cross Abstract: Self-supervised learning enables audio representations that transfer across domains and tasks.

By Ludovic K. Tuncay (IRIT-SAMoVA), Etienne Labb\'e (IRIT-SAMoVA), Thomas Pellegrini (IRIT-SAMoVA)

llms benchmarks

Related stories