arXiv Machine Learning
SpecPV: Improving Self-Speculative Decoding for Long-Context Generation via Partial Verification
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Sep 30
SEED: Self-Speculative Decoding via Implicit Encoder-Decoder
arXiv:2609.36590v1 Announce Type: cross Abstract: Self-speculative decoding accelerates large language model (LLM) inference by drafting tokens from the target model itself, but faces a sharp tradeof...
arXiv Machine Learning
Jun 25
Dustin: Draft-Augmented Sparse Verification for Efficient Long-Context Generation with Speculative Decoding
arXiv:2606. 24957v1 Announce Type: cross Abstract: While speculative decoding improves inference throughput for multi-batch long-context Large Language Models (LLMs), its efficiency is often limited by a verification bottleneck where Key-Value (KV) cache loading dominates latency.
arXiv AI
Sep 30
LongSpark: Efficient speculative decoding with a fixed-cost parallel drafter
arXiv:2609.37029v1 Announce Type: cross Abstract: Speculative decoding accelerates autoregressive inference by verifying multiple draft tokens in a single target forward pass. However, as the context...
arXiv Computation and Language
Sep 2
Beyond Tokens: Semantic-Aware Speculative Decoding for Efficient Inference by Probing Internal States
Large Language Models suffer from high inference latency, especially when generating long chains of thought. Existing speculative decoding methods draft and verify tokens in parallel but ignore semantic equivalence, causing inefficient rejections. The proposed SemanticSpec framework verifies entire semantic sequences by probing internal hidden states, achieving up to 2.7× speedup on DeepSeekR1-32B and 2.1× on QwQ-32B while outperforming token‑level and sequence‑level baselines in both efficiency and effectiveness.
arXiv Machine Learning
Sep 17
ASPIRE: Asynchronous Batched Self-Speculative Decoding for Long-Context LLM Inference
ASPIRE introduces a non‑synchronized batched self‑speculative decoding framework for long‑context LLM inference, addressing the memory bottleneck of attention by drafting tokens with sparse attention and verifying them with full attention. It combines a unified mixed forward pass, a lightweight online speculation scheduler that lets each request independently decide when to verify, and an intra‑draft refresh layer that updates the sparse context at every draft step. Experiments on three models and five benchmarks show 1.70–4.58× speedup over autoregressive baselines and a 27% average improvement over the strongest prior self‑speculative methods.
arXiv AI
Jun 4
SSSD: Simply-Scalable Speculative Decoding
arXiv:2411. 05894v3 Announce Type: replace-cross Abstract: Speculative Decoding has emerged as a popular technique for accelerating inference in Large Language Models.