Speculative Decoding for 2x Faster Whisper Inference

Blazingly fast whisper transcriptions with Inference Endpoints

Introducing Whisper

A Practical Investigation of Training-free Relaxed Speculative Decoding

arXiv:2607. 08690v1 Announce Type: cross Abstract: Speculative decoding accelerates sampling from an autoregressive LLM by using a faster auxiliary model to draft tokens which are then verified in parallel by the LLM.

Faster Text Generation with Self-Speculative Decoding

Whisper-Aware LLM: Self-Supervised Uncertainty Learning for Robust Whispered Speech Recognition

arXiv:2608. 10836v1 Announce Type: cross Abstract: The signal ambiguity of whispered speech drives ASR systems toward two opposing failure modes: failing to capture whispered speech or hallucinatory transcription of noise.

A Practical Investigation of Training-free Relaxed Speculative Decoding

Speculative decoding accelerates sampling from an autoregressive LLM by using a faster auxiliary model to draft tokens which are then verified in parallel by the LLM. Standard speculative decoding is lossless: its rejection and resampling steps exactly preserve the LLM's sampling distribution.

RelayS2S: A Dual-Path Speculative Generation for Real-Time Dialogue

RelayS2S is a hybrid real‑time dialogue system that runs a fast duplex speech‑to‑speech path and a slow ASR‑to‑LLM path in parallel. The fast path speculatively drafts a short response prefix and streams it to TTS, while the slow path generates a higher‑quality continuation conditioned on that prefix. A lightweight verifier decides whether to commit the prefix or fall back to the cascaded pipeline, achieving much lower latency (81 ms P90 first‑chunk) while preserving 99% of the cascaded pipeline’s textual quality.

SSSD: Simply-Scalable Speculative Decoding

arXiv:2411. 05894v3 Announce Type: replace-cross Abstract: Speculative Decoding has emerged as a popular technique for accelerating inference in Large Language Models.

MURMUR: An Efficient Inference System for Long-Form ASR

arXiv:2606. 01483v1 Announce Type: cross Abstract: Long-form automatic speech recognition (ASR) requires both high accuracy and low latency, but existing systems force a trade-off between the two.

Powerful ASR + diarization + speculative decoding with Hugging Face Inference Endpoints

Revisiting Lossy Verification in Speculative Decoding: Mechanisms, Trade-offs, and Failure Modes

Speculative Decoding (SD) accelerates large language model inference by allowing a lightweight draft model to propose tokens that are subsequently verified in parallel by a larger target model. Recent approaches introduce lossy verification schemes to further improve efficiency by relaxing strict distributional matching.

Dustin: Draft-Augmented Sparse Verification for Efficient Long-Context Generation with Speculative Decoding

arXiv:2606. 24957v1 Announce Type: cross Abstract: While speculative decoding improves inference throughput for multi-batch long-context Large Language Models (LLMs), its efficiency is often limited by a verification bottleneck where Key-Value (KV) cache loading dominates latency.