arXiv Machine Learning

Geometric Iterative Retrieval for Neural Audio Codec Resynthesis

The paper introduces geometric iterative retrieval, a new approach for resynthesizing high‑quality audio from coarse Residual Vector Quantization (RVQ) codec tokens. Instead of choosing between discrete token prediction or continuous regression, the method performs contrastive retrieval within the continuous codebook space, leveraging the RVQ hierarchy as an iterative decomposition. Experiments on speech and music codec restoration tasks demonstrate that this technique outperforms both single‑pass token prediction and one‑step regression baselines.

Hugging Face Trending Papers

Jun 3

CleanCodec: Efficient and Robust Speech Tokenization via Perceptually Guided Encoding

Neural audio codecs are a key component of speech processing pipelines, compressing audio into discrete tokens for downstream modeling. However, existing codecs struggle to balance reconstruction quality with token efficiency, often encoding perceptually irrelevant information such as background noise and recording artifacts at the expense of linguistically and acoustically meaningful content.

nlp multimodal benchmarks

arXiv AI

Jun 3

EntangleCodec: A Unified Discrete Audio Tokenizer via Semantic-Acoustic Entanglement

arXiv:2606. 02739v1 Announce Type: cross Abstract: Audio tokenizers serve as the discrete interface between continuous audio and Audio Language Models (ALMs), but existing tokenizers often struggle to support both understanding and generation.

By Hui Li, Yangfan Gao, Junlin Shang, Changhao Jiang, Tao Gui, Qi Zhang, Xuanjing Huang

llms diffusion nlp efficiency multimodal benchmarks

arXiv AI

Jun 6

F3-Tokenizer: Taming Audio Autoencoder Latents for Understanding and Generation

arXiv:2606. 06357v1 Announce Type: cross Abstract: Continuous audio autoencoders reconstruct waveforms well but often produce latents with weak structure for understanding, while self-supervised audio encoders capture semantics but are not directly decodable.

By Dinghao Zhou, Xingchen Song, Di Wu, Pengyu Cheng, Shengfan Shen, Sixiang Lv

llms nlp

arXiv AI

Jun 8

HybridCodec: Fast Dual-Stream, Semantically Enhanced Neural Audio Codec

arXiv:2606. 06743v1 Announce Type: cross Abstract: The popularity of neural audio codecs as speech tokenizers has surged with the advent of Multimodal Large Language Models.

By Arjun Gangwar, S Umesh

llms multimodal

arXiv Machine Learning

Jun 9

PairAlign: A Framework for Sequence Tokenization via Self-Alignment with Applications to Audio Tokenization

arXiv:2605. 06582v2 Announce Type: replace Abstract: Many operations on sensory data -- comparison, memory, retrieval, and reasoning -- are naturally expressed over discrete symbolic structures.

By Adhiraj Banerjee, Vipul Arora

nlp efficiency safety

arXiv AI

Aug 3

Stable Autoregressive Speech Generation with Low-Frame-Rate High-Dimensional Continuous Tokens

arXiv:2607. 29363v1 Announce Type: cross Abstract: Balancing sequence length, representational capacity, and long-horizon stability is a central problem in autoregressive (AR) speech and audio generation.

By Yi Luo, Rongzhi Gu, Jixun Yao

llms multimodal

arXiv AI

Jun 29

HybridCodec: Modeling Discrete and Continuous Representations for Efficient Speech Language Models

arXiv:2606. 27627v1 Announce Type: cross Abstract: Discrete audio representations have become increasingly popular for building multimodal text-audio systems and integrating audio capabilities into Large Language Models (LLMs).

By Artem Ploujnikov, Francesco Verdini, Samir Sadok, Mirco Ravanelli

llms multimodal

arXiv AI

Jul 28

The Equalizer: Introducing Shape-Gain Decomposition in Neural Audio Codecs

arXiv:2602. 15491v2 Announce Type: replace-cross Abstract: Neural audio codecs (NACs) typically encode the short-term energy (gain) and normalized structure (shape) of speech/audio signals jointly within the same latent space.

By Samir Sadok, Laurent Girin, Xavier Alameda-Pineda

rag diffusion efficiency

arXiv AI

Jul 7

Shao: Scaling Acoustic Token Language Models Toward High-Fidelity Music Generation

arXiv:2605. 01790v2 Announce Type: replace-cross Abstract: A common design pattern in high-quality music generation is to handle structure and fidelity in different representation spaces: a generator first models high-level structure, followed by diffusion-based or neural decoding stages that reconstruct fine details.

By Jiafeng Liu, Yuanliang Dong, Hongjia Liu, Yuqing Cheng, Zhancheng Guo, Huijing Liang, Wenbo Zhan, Yuming Sun, Xiaobing Li, Feng Yu, Maosong Sun

llms diffusion efficiency safety

arXiv AI

Jul 7

Probing Low-Level Acoustic Attribute Encoding in CLAP Audio Embeddings

arXiv:2607. 03806v1 Announce Type: cross Abstract: Audio foundation models are widely adopted as general-purpose feature extractors, yet the internal structure of their learned representations remains insufficiently understood.

By H\'ector Martel, Joe Hennessy-Priest, Taemin Cho

rag multimodal

arXiv AI

Jun 10

Whisper-GPT -- Continuous Discrete Hybrid Representation Language Models For Speech And Music

arXiv:2412. 11449v2 Announce Type: replace-cross Abstract: We propose WHISPER-GPT: A generative large language model (LLM) for speech and music that allows us to work with continuous audio representations and discrete tokens simultaneously as part of a single architecture.

By Prateek Verma

llms multimodal

arXiv Machine Learning

Jul 2

Evaluating Pretrained Music Embeddings for Cross-Performance Jazz Standard Recognition

arXiv:2607. 00777v1 Announce Type: cross Abstract: Recognizing jazz standards from audio is a challenging form of tune-level music retrieval: different performances of the same standard may vary in tempo, key, arrangement, instrumentation, improvisational content, and even whether the head melody is present.

By \c{C}a\u{g}r{\i} Eser

rag

Related stories