ZipCodec is a streaming neural speech codec that operates at an ultra‑low frame rate of 6.25 Hz and a bitrate of 0.80 kbps, achieving a theoretical latency of 160 ms. It leverages large‑scale WavLM distillation, a redesigned transformer architecture, a scalar spherical quantizer, and a latency‑aware streaming decoder to preserve reconstruction quality while reducing frame rate. Experiments demonstrate that ZipCodec outperforms existing streaming codecs at comparable bitrates in both reconstruction and downstream tasks, and it can run real‑time single‑stream inference on a consumer‑grade CPU despite having 842 M parameters.
By Luca Della Libera, Cem Subakan, Mirco Ravanelli
arXiv:2603.05887v2 Announce Type: replace-cross
Abstract: Neural audio codecs optimized for mel-spectrogram reconstruction often fail to preserve intelligibility. While semantic encoder distillation...
By Junhyeok Lee, Xiluo He, Jihwan Lee, Helin Wang, Shrikanth Narayanan, Thomas Thebaud, Laureano Moro-Velazquez, Jes\'us Villalba, Najim Dehak
Neural audio codecs are a key component in speech language modeling. However, their high frame rates lead to long sequence lengths, increasing computational costs. Dynamic frame rate codecs mitigate t...
arXiv:2610.15453v1 Announce Type: cross
Abstract: Neural audio codecs achieve remarkable performance but operate as closed ecosystems with rigidly co-trained encoderdecoder pairs, severely hindering...
By Zijian Lin, Jing Yang, Jinghao Luo, Zhuo Wang, Fan Fan, Zhiyong Wu
arXiv:2609.17509v1 Announce Type: cross
Abstract: Neural audio codecs are a key component in speech language modeling. However, their high frame rates lead to long sequence lengths, increasing comput...
By Thanapat Trachu, Samuele Cornell, William Chen, Shinji Watanabe
The paper introduces a memory‑efficient audio synthesis architecture used in Siri Expressive Voices, converting semantic audio tokens into high‑fidelity audio via a residual vector quantization (RVQ) pipeline that decouples temporal and depth processing. A single Diffusion Transformer‑style depth decoder generates all RVQ levels autoregressively, while causal sliding‑window attention with fixed‑window key‑value caching keeps memory usage constant regardless of sequence length. Deployed on Apple’s AMX, the system achieves real‑time performance (≈10 ms per step) with only 21 MB peak runtime memory and 329 MB of on‑device assets, enabling continuous streaming of 20–320 s of audio and improving Mean Opinion Scores by +0.28 overall and +0.42 on conversational speech compared to the previous on‑device TTS system.
By Dongseong Hwang, Prasanth Yadla, Kaan Elgin, Shifas Padinjaru Veettil, Sivanand Achanta, Dipjyoti Paul, Ramya Rasipuram, Tyler Johnson, Emad Soroush, Chung-Cheng Chiu, Zhifeng Chen
arXiv:2603. 08683v2 Announce Type: replace-cross Abstract: Autoregressive "language" models (LMs) trained on raw waveforms can be repurposed for lossless audio compression, but prior work is limited to 8-bit audio, leaving open whether such approaches work for practical settings (16/24-bit) and can compete with existing codecs.
By Phillip Long, Zachary Novack, Chris Donahue
arXiv:2608. 05727v1 Announce Type: cross Abstract: Neural Audio Codecs are widely adopted in speech generation and editing.
By June Young Yi, Dongwook Lee, Jiheum Yeom, Sungroh Yoon
arXiv:2512.21653v2 Announce Type: replace-cross
Abstract: Speech codecs are usually optimized for waveform fidelity, allocating bits to acoustic detail that can be inferred from linguistic structure....
By Liuyang Bai, Weiyi Lu, Li Guo
X-VC is a zero‑shot streaming voice conversion system that performs one‑step conversion directly in the latent space of a pretrained neural codec. It employs a dual‑conditioning acoustic converter that jointly models source codec latents and target acoustic conditions, while using adaptive normalization to inject utterance‑level speaker information. The model is trained with generated paired data and a role‑assignment strategy, and uses a chunkwise inference scheme with overlap smoothing to achieve low‑latency streaming inference, achieving superior WER, speaker similarity, and real‑time factor on the Seed‑TTS‑Eval benchmark.
By Qixi Zheng, Yuxiang Zhao, Tianrui Wang, Wenxi Chen, Kele Xu, Yikang Li, Qinyuan Cheng, Xipeng Qiu, Kai Yu, Xie Chen
arXiv:2606. 06743v1 Announce Type: cross Abstract: The popularity of neural audio codecs as speech tokenizers has surged with the advent of Multimodal Large Language Models.
By Arjun Gangwar, S Umesh
TontaubeV1 is a streaming text‑to‑speech model that maintains natural prosody while running on a single consumer GPU. It encodes speech with a hierarchical DualCodec representation at 12.5 Hz, separating a semantic stream from successive acoustic refinements, and uses Qwen3‑derived transformers to predict the semantic stream and add refinements. The model supports up to one minute of reference audio for voice conditioning, streams with a 200 ms latency to first audio, and achieves real‑time factors of 0.08 (single input) and 0.02 (eight concurrent inputs).