ZipCodec is a streaming neural speech codec that operates at an ultra‑low frame rate of 6.25 Hz and a bitrate of 0.80 kbps, achieving a theoretical latency of 160 ms. It leverages large‑scale WavLM distillation, a redesigned transformer architecture, a scalar spherical quantizer, and a latency‑aware streaming decoder. Experiments demonstrate that ZipCodec outperforms existing streaming codecs at comparable bitrates in both reconstruction quality and downstream tasks, while remaining real‑time on a consumer‑grade CPU despite its 842 M parameters.
arXiv:2603.05887v2 Announce Type: replace-cross
Abstract: Neural audio codecs optimized for mel-spectrogram reconstruction often fail to preserve intelligibility. While semantic encoder distillation...
By Junhyeok Lee, Xiluo He, Jihwan Lee, Helin Wang, Shrikanth Narayanan, Thomas Thebaud, Laureano Moro-Velazquez, Jes\'us Villalba, Najim Dehak
arXiv:2609.17509v1 Announce Type: cross
Abstract: Neural audio codecs are a key component in speech language modeling. However, their high frame rates lead to long sequence lengths, increasing comput...
By Thanapat Trachu, Samuele Cornell, William Chen, Shinji Watanabe
arXiv:2610.15453v1 Announce Type: cross
Abstract: Neural audio codecs achieve remarkable performance but operate as closed ecosystems with rigidly co-trained encoderdecoder pairs, severely hindering...
By Zijian Lin, Jing Yang, Jinghao Luo, Zhuo Wang, Fan Fan, Zhiyong Wu
Neural audio codecs are a key component in speech language modeling. However, their high frame rates lead to long sequence lengths, increasing computational costs. Dynamic frame rate codecs mitigate t...
The paper introduces a memory‑efficient audio synthesis architecture used in Siri Expressive Voices, converting semantic audio tokens into high‑fidelity audio via a residual vector quantization (RVQ) pipeline that decouples temporal and depth processing. A single Diffusion Transformer‑style depth decoder generates all RVQ levels autoregressively, while causal sliding‑window attention with fixed‑window key‑value caching keeps memory usage constant regardless of sequence length. Deployed on Apple’s AMX, the system achieves real‑time performance (≈10 ms per step) with only 21 MB peak runtime memory and 329 MB of on‑device assets, enabling continuous streaming of 20–320 s of audio and improving Mean Opinion Scores by +0.28 overall and +0.42 on conversational speech compared to the previous on‑device TTS system.
By Dongseong Hwang, Prasanth Yadla, Kaan Elgin, Shifas Padinjaru Veettil, Sivanand Achanta, Dipjyoti Paul, Ramya Rasipuram, Tyler Johnson, Emad Soroush, Chung-Cheng Chiu, Zhifeng Chen