arXiv Machine Learning

Geometric Iterative Retrieval for Neural Audio Codec Resynthesis

The paper introduces geometric iterative retrieval, a new approach for resynthesizing high‑quality audio from coarse Residual Vector Quantization (RVQ) codec tokens. Instead of choosing between discrete token prediction or continuous regression, the method performs contrastive retrieval within the continuous codebook space, leveraging the RVQ hierarchy as an iterative decomposition. Experiments on speech and music codec restoration tasks demonstrate that this technique outperforms both single‑pass token prediction and one‑step regression baselines.

Hugging Face Trending Papers

Jun 3

CleanCodec: Efficient and Robust Speech Tokenization via Perceptually Guided Encoding

Neural audio codecs are a key component of speech processing pipelines, compressing audio into discrete tokens for downstream modeling. However, existing codecs struggle to balance reconstruction quality with token efficiency, often encoding perceptually irrelevant information such as background noise and recording artifacts at the expense of linguistically and acoustically meaningful content.

nlp multimodal benchmarks

arXiv AI

Sep 16

LACE: Layer-Wise Compression for Dynamic Frame Rate Codecs

arXiv:2609.17509v1 Announce Type: cross Abstract: Neural audio codecs are a key component in speech language modeling. However, their high frame rates lead to long sequence lengths, increasing comput...

By Thanapat Trachu, Samuele Cornell, William Chen, Shinji Watanabe

rag computer-vision efficiency multimodal safety

arXiv AI

Sep 15

LoSATok: Low-Dimensional Semantic-Acoustic Tokenizer for Cross-Domain Audio Understanding and Generation

arXiv:2605.27840v2 Announce Type: replace-cross Abstract: Audio tokenizers are fundamental to unifying audio understanding and generation. Understanding requires high-level semantics, while generatio...

By Zhisheng Zhang, Xiang Li, Yixuan Zhou, Jing Peng, Guoyang Zeng, Zhiyong Wu

llms diffusion nlp

arXiv AI

Jun 3

EntangleCodec: A Unified Discrete Audio Tokenizer via Semantic-Acoustic Entanglement

arXiv:2606. 02739v1 Announce Type: cross Abstract: Audio tokenizers serve as the discrete interface between continuous audio and Audio Language Models (ALMs), but existing tokenizers often struggle to support both understanding and generation.

By Hui Li, Yangfan Gao, Junlin Shang, Changhao Jiang, Tao Gui, Qi Zhang, Xuanjing Huang

llms diffusion nlp efficiency multimodal benchmarks

arXiv Machine Learning

Aug 27

BRIDLE: Generalized Self-supervised Learning with Quantization

BRIDLE is a self‑supervised encoder pretraining framework that extends bidirectional training to audio, image, and video by incorporating residual quantization (RQ) with multiple hierarchical codebooks. This approach allows fine‑grained discretization of latent representations and interleaves training between the encoder and tokenizer. Experiments show that BRIDLE achieves state‑of‑the‑art results on audio classification benchmarks and competitive performance on image and video classification tasks, outperforming traditional vector‑quantization methods.

By Hoang M. Nguyen, Satya N. Shukla, Qiang Zhang, Hanchao Yu, Sreya D. Roy, Dipesh Tamboli, Taipeng Tian, Lingjiong Zhu, Yuchen Liu

diffusion computer-vision nlp efficiency benchmarks

Hugging Face Trending Papers

Sep 15

LACE: Layer-Wise Compression for Dynamic Frame Rate Codecs

Neural audio codecs are a key component in speech language modeling. However, their high frame rates lead to long sequence lengths, increasing computational costs. Dynamic frame rate codecs mitigate t...

rag computer-vision efficiency multimodal safety

arXiv AI

Jun 6

F3-Tokenizer: Taming Audio Autoencoder Latents for Understanding and Generation

arXiv:2606. 06357v1 Announce Type: cross Abstract: Continuous audio autoencoders reconstruct waveforms well but often produce latents with weak structure for understanding, while self-supervised audio encoders capture semantics but are not directly decodable.

By Dinghao Zhou, Xingchen Song, Di Wu, Pengyu Cheng, Shengfan Shen, Sixiang Lv

llms nlp

arXiv AI

Jun 8

HybridCodec: Fast Dual-Stream, Semantically Enhanced Neural Audio Codec

arXiv:2606. 06743v1 Announce Type: cross Abstract: The popularity of neural audio codecs as speech tokenizers has surged with the advent of Multimodal Large Language Models.

By Arjun Gangwar, S Umesh

llms multimodal

arXiv Machine Learning

Jun 9

PairAlign: A Framework for Sequence Tokenization via Self-Alignment with Applications to Audio Tokenization

arXiv:2605. 06582v2 Announce Type: replace Abstract: Many operations on sensory data -- comparison, memory, retrieval, and reasoning -- are naturally expressed over discrete symbolic structures.

By Adhiraj Banerjee, Vipul Arora

nlp efficiency safety

arXiv AI

Aug 3

Stable Autoregressive Speech Generation with Low-Frame-Rate High-Dimensional Continuous Tokens

arXiv:2607. 29363v1 Announce Type: cross Abstract: Balancing sequence length, representational capacity, and long-horizon stability is a central problem in autoregressive (AR) speech and audio generation.

By Yi Luo, Rongzhi Gu, Jixun Yao

llms multimodal

arXiv AI

Jun 29

HybridCodec: Modeling Discrete and Continuous Representations for Efficient Speech Language Models

arXiv:2606. 27627v1 Announce Type: cross Abstract: Discrete audio representations have become increasingly popular for building multimodal text-audio systems and integrating audio capabilities into Large Language Models (LLMs).

By Artem Ploujnikov, Francesco Verdini, Samir Sadok, Mirco Ravanelli

llms multimodal

arXiv AI

Jul 28

The Equalizer: Introducing Shape-Gain Decomposition in Neural Audio Codecs

arXiv:2602. 15491v2 Announce Type: replace-cross Abstract: Neural audio codecs (NACs) typically encode the short-term energy (gain) and normalized structure (shape) of speech/audio signals jointly within the same latent space.

By Samir Sadok, Laurent Girin, Xavier Alameda-Pineda

rag diffusion efficiency

Related stories