arXiv Machine Learning
Aug 7
LILAC: An Idempotent Neural Speech Codec
arXiv:2608. 05727v1 Announce Type: cross Abstract: Neural Audio Codecs are widely adopted in speech generation and editing.
CleanCodec: Efficient and Robust Speech Tokenization via Perceptually Guided Encoding
Neural audio codecs are a key component of speech processing pipelines, compressing audio into discrete tokens for downstream modeling. However, existing codecs struggle to balance reconstruction quality with token efficiency, often encoding perceptually irrelevant information such as background noise and recording artifacts at the expense of linguistically and acoustically meaningful content.
Related stories
arXiv Machine Learning
Sep 14
TokenMapper: A Step Toward Interoperable Speech Token Translation
TokenMapper is a framework that enables direct translation between different speech tokenizers, allowing heterogeneous speech models to communicate without converting tokens to waveform audio. It handles mismatched token spaces, including single and multi-codebook representations, while maintaining a shared effective token rate. Experiments on GLM-4-Voice, MiMi, and DualCodec show that TokenMapper achieves word error rates close to native reconstructions, comparable human MOS scores, and significantly reduces latency compared to waveform bridging.
arXiv AI
Jun 3
EntangleCodec: A Unified Discrete Audio Tokenizer via Semantic-Acoustic Entanglement
arXiv:2606. 02739v1 Announce Type: cross Abstract: Audio tokenizers serve as the discrete interface between continuous audio and Audio Language Models (ALMs), but existing tokenizers often struggle to support both understanding and generation.
arXiv AI
Aug 21
Towards Audio Token Compression in Large Audio Language Models
arXiv:2511. 20973v2 Announce Type: replace-cross Abstract: Large Audio Language Models (LALMs) deliver strong performance across speech and audio tasks, but their audio encoders generate high-rate token sequences (e.
Hugging Face Trending Papers
Jul 6
Unified Audio Intelligence Without Regressing on Text Intelligence
Audio intelligence involves understanding, reasoning about, and generating both audio and speech. In this work, we introduce Nemotron-Labs-Audex-30B-A3B (Audex), a unified audio-text LLM built on Nemotron-Cascade-2-30B-A3B, a strong text-only MoE LLM.
arXiv AI
Jun 2
DSA-Tokenizer: Disentangled Semantic-Acoustic Tokenization via Flow Matching-based Hierarchical Fusion
arXiv:2601. 09239v5 Announce Type: replace-cross Abstract: Speech tokenizers are a key building block of fully discrete Speech LLMs.
arXiv AI
Jun 10
Whisper-GPT -- Continuous Discrete Hybrid Representation Language Models For Speech And Music
arXiv:2412. 11449v2 Announce Type: replace-cross Abstract: We propose WHISPER-GPT: A generative large language model (LLM) for speech and music that allows us to work with continuous audio representations and discrete tokens simultaneously as part of a single architecture.
arXiv AI
Jun 8
HybridCodec: Fast Dual-Stream, Semantically Enhanced Neural Audio Codec
arXiv:2606. 06743v1 Announce Type: cross Abstract: The popularity of neural audio codecs as speech tokenizers has surged with the advent of Multimodal Large Language Models.
Hugging Face Trending Papers
Jun 10
Feature-Aligned Speech Watermarking for Robustness to Reconstruction Distortions
Audio watermarking aims to embed identifiable information into audio while remaining imperceptible. Existing methods adopt high-fidelity, low-energy designs to preserve perceptual quality, but the resulting watermarks lack robustness under suppression by speech reconstruction models.
arXiv Machine Learning
Sep 10
Decoder-Side Semantic Conditioning for Low-Bitrate Neural Speech Compression
arXiv:2512.21653v2 Announce Type: replace-cross Abstract: Speech codecs are usually optimized for waveform fidelity, allocating bits to acoustic detail that can be inferred from linguistic structure....
arXiv AI
Jun 6
F3-Tokenizer: Taming Audio Autoencoder Latents for Understanding and Generation
arXiv:2606. 06357v1 Announce Type: cross Abstract: Continuous audio autoencoders reconstruct waveforms well but often produce latents with weak structure for understanding, while self-supervised audio encoders capture semantics but are not directly decodable.
arXiv Machine Learning
Aug 20
Geometric Iterative Retrieval for Neural Audio Codec Resynthesis
The paper introduces geometric iterative retrieval, a new approach for resynthesizing high‑quality audio from coarse Residual Vector Quantization (RVQ) codec tokens. Instead of choosing between discrete token prediction or continuous regression, the method performs contrastive retrieval within the continuous codebook space, leveraging the RVQ hierarchy as an iterative decomposition. Experiments on speech and music codec restoration tasks demonstrate that this technique outperforms both single‑pass token prediction and one‑step regression baselines.