ZipCodec is a streaming neural speech codec that operates at an ultra‑low frame rate of 6.25 Hz and a bitrate of 0.80 kbps, achieving a theoretical latency of 160 ms. It leverages large‑scale WavLM distillation, a redesigned transformer architecture, a scalar spherical quantizer, and a latency‑aware streaming decoder. Experiments demonstrate that ZipCodec outperforms existing streaming codecs at comparable bitrates in both reconstruction quality and downstream tasks, while remaining real‑time on a consumer‑grade CPU despite its 842 M parameters.
arXiv:2603.05887v2 Announce Type: replace-cross
Abstract: Neural audio codecs optimized for mel-spectrogram reconstruction often fail to preserve intelligibility. While semantic encoder distillation...
By Junhyeok Lee, Xiluo He, Jihwan Lee, Helin Wang, Shrikanth Narayanan, Thomas Thebaud, Laureano Moro-Velazquez, Jes\'us Villalba, Najim Dehak
arXiv:2609.17509v1 Announce Type: cross
Abstract: Neural audio codecs are a key component in speech language modeling. However, their high frame rates lead to long sequence lengths, increasing comput...
By Thanapat Trachu, Samuele Cornell, William Chen, Shinji Watanabe
arXiv:2610.15453v1 Announce Type: cross
Abstract: Neural audio codecs achieve remarkable performance but operate as closed ecosystems with rigidly co-trained encoderdecoder pairs, severely hindering...
By Zijian Lin, Jing Yang, Jinghao Luo, Zhuo Wang, Fan Fan, Zhiyong Wu
Neural audio codecs are a key component in speech language modeling. However, their high frame rates lead to long sequence lengths, increasing computational costs. Dynamic frame rate codecs mitigate t...
The paper introduces a memory‑efficient audio synthesis architecture used in Siri Expressive Voices, converting semantic audio tokens into high‑fidelity audio via a residual vector quantization (RVQ) pipeline that decouples temporal and depth processing. A single Diffusion Transformer‑style depth decoder generates all RVQ levels autoregressively, while causal sliding‑window attention with fixed‑window key‑value caching keeps memory usage constant regardless of sequence length. Deployed on Apple’s AMX, the system achieves real‑time performance (≈10 ms per step) with only 21 MB peak runtime memory and 329 MB of on‑device assets, enabling continuous streaming of 20–320 s of audio and improving Mean Opinion Scores by +0.28 overall and +0.42 on conversational speech compared to the previous on‑device TTS system.
By Dongseong Hwang, Prasanth Yadla, Kaan Elgin, Shifas Padinjaru Veettil, Sivanand Achanta, Dipjyoti Paul, Ramya Rasipuram, Tyler Johnson, Emad Soroush, Chung-Cheng Chiu, Zhifeng Chen
arXiv:2512.21653v2 Announce Type: replace-cross
Abstract: Speech codecs are usually optimized for waveform fidelity, allocating bits to acoustic detail that can be inferred from linguistic structure....
By Liuyang Bai, Weiyi Lu, Li Guo
arXiv:2603. 08683v2 Announce Type: replace-cross Abstract: Autoregressive "language" models (LMs) trained on raw waveforms can be repurposed for lossless audio compression, but prior work is limited to 8-bit audio, leaving open whether such approaches work for practical settings (16/24-bit) and can compete with existing codecs.
By Phillip Long, Zachary Novack, Chris Donahue
arXiv:2608.30927v1 Announce Type: cross
Abstract: Whisper exposes speech through a fixed 1500-token encoder interface, now a default representation for ASR decoders and Whisper-based speech language...
By Chanhee Cho, Junhyuk Choi, Bugeun Kim
arXiv:2608. 08569v1 Announce Type: new Abstract: Recent advancements in Speech Large Language Models have demonstrated remarkable capabilities in understanding complex audio tasks.
By Wenxu Jia, Dongjie Fu, Xize Cheng, Fangming Feng, Linjun Li, Wenshi Chen, Yingming Li, Zhou Zhao, Tao Jin
arXiv:2606. 06743v1 Announce Type: cross Abstract: The popularity of neural audio codecs as speech tokenizers has surged with the advent of Multimodal Large Language Models.
By Arjun Gangwar, S Umesh
X-VC is a zero‑shot streaming voice conversion system that performs one‑step conversion directly in the latent space of a pretrained neural codec. It employs a dual‑conditioning acoustic converter that jointly models source codec latents and target acoustic conditions, while using adaptive normalization to inject utterance‑level speaker information. The model is trained with generated paired data and a role‑assignment strategy, and uses a chunkwise inference scheme with overlap smoothing to achieve low‑latency streaming inference, achieving superior WER, speaker similarity, and real‑time factor on the Seed‑TTS‑Eval benchmark.
By Qixi Zheng, Yuxiang Zhao, Tianrui Wang, Wenxi Chen, Kele Xu, Yikang Li, Qinyuan Cheng, Xipeng Qiu, Kai Yu, Xie Chen