GeoRVQ introduces a coarse‑to‑fine masked token model that incorporates decoder‑induced geometry into residual‑vector‑quantized physiological waveforms. By using geometry‑aware soft targets and expected distortion, the model improves exact token accuracy from 0.133 to 0.143, reduces decoded distance from 0.606 to 0.393, and raises R‑peak F1 from 0.784 to 0.837 on datasets such as MIMIC‑IV Waveform, VitalDB, and CODE‑15%. The approach demonstrates that decoder‑aware objectives can enhance waveform and event preservation without a large increase in token accuracy.
arXiv:2606. 02631v1 Announce Type: cross Abstract: This paper studies whether audio, images, and video can share a common wavelet token schema rather than relying on separate modality-specific latent grids.
arXiv:2606. 09605v1 Announce Type: new Abstract: Foundation models offer a promising route to compress multi-modal physiological signals into compact representations of human health, with broad applications across sleep medicine, cardiology, neurology and other healthcare domains.
arXiv:2512.21653v2 Announce Type: replace-cross Abstract: Speech codecs are usually optimized for waveform fidelity, allocating bits to acoustic detail that can be inferred from linguistic structure....
The paper introduces Triage, a method that predicts the attention distribution of audio tokens before a language model processes them, enabling early pruning of less important tokens. By fitting a linear map to encoder outputs, Triage achieves high correlation (ρ ≥ 0.69) with full-model attention across eleven of thirteen large audio language models. Using this prediction, Triage compresses audio inputs while maintaining near‑full performance, outperforming baselines in transcription accuracy and significantly increasing the amount of audio that fits within a model’s context window.
TokenMapper is a framework that enables direct translation between different speech tokenizers, allowing heterogeneous speech models to communicate without converting tokens to waveform audio. It handles mismatched token spaces, including single and multi-codebook representations, while maintaining a shared effective token rate. Experiments on GLM-4-Voice, MiMi, and DualCodec show that TokenMapper achieves word error rates close to native reconstructions, comparable human MOS scores, and significantly reduces latency compared to waveform bridging.
The paper introduces a method to prune six layers from the encoder of OpenAI’s Whisper ASR model, reducing the encoder stack by 18.5% without requiring custom inference code. Layers are selected based on their minimal impact on Word Error Rate when removed. After pruning, the model’s WER rises from 18.2% to 21.9%, but distillation with unlabeled monolingual speech data lowers it to 20.1%. "whyItMatters":"The approach offers a straightforward way to accelerate Whisper inference by simplifying the encoder while maintaining acceptable accuracy, and the released code and model enable immediate adoption by the community."
arXiv:2609.36687v1 Announce Type: cross Abstract: Neural codecs encode continuous signals into compact sequences of discrete tokens, providing an interface for efficient transmission, storage, and to...
Pruning large pre-trained transformer-based ASR models such as OpenAI's Whisper has seen great adoption, as pruning the decoder led to significant end-to-end transcription speedups. For instance, the...
The paper investigates how encoder-decoder models in ASR and NMT can generate fluent text even when the input contains no recoverable message, a phenomenon known as message-free hallucination. By auditing the models’ reserved null tokens and manipulating their scores, the authors show that a higher null-token score can suppress fabrication but may also delete valid content or shorten translations. The study highlights that the null token can serve as a diagnostic tool for hallucination and suggests evaluating abstention methods by considering both suppression and deletion costs.
arXiv:2606. 09019v1 Announce Type: cross Abstract: Codec-based autoregressive (AR) speech language models have achieved strong text-to-speech (TTS) quality by modeling speech as sequences of discrete audio tokens with large pretrained backbones.
arXiv:2608. 08569v1 Announce Type: new Abstract: Recent advancements in Speech Large Language Models have demonstrated remarkable capabilities in understanding complex audio tasks.
The paper introduces a post‑training softmax reparameterization technique that selects a functionally equivalent output head before quantization. By subtracting a scalar multiple of the vocabulary‑row mean from each output row and tuning this coefficient via validation KL, the method preserves the full‑precision softmax distribution while enabling efficient W4 quantization. Experiments on seven heads show significant error reductions and latency improvements, with the approach remaining complementary to other quantization strategies and transferable across datasets.