Neural audio codecs are a key component of speech processing pipelines, compressing audio into discrete tokens for downstream modeling. However, existing codecs struggle to balance reconstruction quality with token efficiency, often encoding perceptually irrelevant information such as background noise and recording artifacts at the expense of linguistically and acoustically meaningful content.
arXiv:2603.05887v2 Announce Type: replace-cross
Abstract: Neural audio codecs optimized for mel-spectrogram reconstruction often fail to preserve intelligibility. While semantic encoder distillation...
By Junhyeok Lee, Xiluo He, Jihwan Lee, Helin Wang, Shrikanth Narayanan, Thomas Thebaud, Laureano Moro-Velazquez, Jes\'us Villalba, Najim Dehak
arXiv:2606. 06743v1 Announce Type: cross Abstract: The popularity of neural audio codecs as speech tokenizers has surged with the advent of Multimodal Large Language Models.
By Arjun Gangwar, S Umesh
ZipCodec is a streaming neural speech codec that operates at an ultra‑low frame rate of 6.25 Hz and a bitrate of 0.80 kbps, achieving a theoretical latency of 160 ms. It leverages large‑scale WavLM distillation, a redesigned transformer architecture, a scalar spherical quantizer, and a latency‑aware streaming decoder. Experiments demonstrate that ZipCodec outperforms existing streaming codecs at comparable bitrates in both reconstruction quality and downstream tasks, while remaining real‑time on a consumer‑grade CPU despite its 842 M parameters.
arXiv:2606. 16969v1 Announce Type: cross Abstract: Low frame rates in neural audio codecs are attractive for autoregressive speech synthesis, where the generation cost scales linearly with the sequence length.
By Alex Gichamba, Moise Busogi
arXiv:2512.21653v2 Announce Type: replace-cross
Abstract: Speech codecs are usually optimized for waveform fidelity, allocating bits to acoustic detail that can be inferred from linguistic structure....
By Liuyang Bai, Weiyi Lu, Li Guo
ZipCodec is a streaming neural speech codec that operates at an ultra‑low frame rate of 6.25 Hz and a bitrate of 0.80 kbps, achieving a theoretical latency of 160 ms. It leverages large‑scale WavLM distillation, a redesigned transformer architecture, a scalar spherical quantizer, and a latency‑aware streaming decoder to preserve reconstruction quality while reducing frame rate. Experiments demonstrate that ZipCodec outperforms existing streaming codecs at comparable bitrates in both reconstruction and downstream tasks, and it can run real‑time single‑stream inference on a consumer‑grade CPU despite having 842 M parameters.
By Luca Della Libera, Cem Subakan, Mirco Ravanelli
PitchFlower is a flow‑based neural audio codec that offers explicit pitch controllability by flattening and randomly shifting F0 contours during training while conditioning on the true F0 to reconstruct the original audio. A vector‑quantization bottleneck blocks pitch recovery, and a flow‑based decoder produces high‑quality audio. Experiments demonstrate that PitchFlower matches DSP baselines in pitch accuracy, surpasses state‑of‑the‑art neural codecs in audio quality, and remains robust even when trained on WORLD‑transformed audio, effectively removing vocoder artifacts.
By Diego Torres, Axel Roebel, Nicolas Obin
The paper introduces Masked Autoregressive Speech Enhancement (MARSE), a method that iteratively decodes masked clean speech frames using continuous latent representations from a neural audio codec (DAC). Unlike prior approaches that relied on discrete token representations, MARSE employs a Conformer model and explores various decoding policies to balance speech enhancement performance with computational cost. The authors provide audio examples and code online to demonstrate the method’s effectiveness.
By Yoto Fujita, Simon Leglaive, Laurent Girin
arXiv:2412. 11449v2 Announce Type: replace-cross Abstract: We propose WHISPER-GPT: A generative large language model (LLM) for speech and music that allows us to work with continuous audio representations and discrete tokens simultaneously as part of a single architecture.
By Prateek Verma
TokenMapper is a framework that enables direct translation between different speech tokenizers, allowing heterogeneous speech models to communicate without converting tokens to waveform audio. It handles mismatched token spaces, including single and multi-codebook representations, while maintaining a shared effective token rate. Experiments on GLM-4-Voice, MiMi, and DualCodec show that TokenMapper achieves word error rates close to native reconstructions, comparable human MOS scores, and significantly reduces latency compared to waveform bridging.
By Tal Kozakov, Tal Rosenwein, Eliya Nachmani
arXiv:2609.36687v1 Announce Type: cross
Abstract: Neural codecs encode continuous signals into compact sequences of discrete tokens, providing an interface for efficient transmission, storage, and to...
By Jihwan Lee, Kleanthis Avramidis, Junhyeok Lee, Tiantian Feng, Najim Dehak, Shrikanth Narayanan