arXiv Machine Learning
H-Spec: Parallel Speculative Decoding Without a Drafter-Side KV Cache
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Computation and Language
Oct 12
Contextual Correction of Parallel Draftsfor Speculative Decoding
The paper introduces DCOCO (DFlash Contextual Corrector), a lightweight mechanism that restores intra-block dependencies in speculative decoding by evaluating multiple candidate tokens for contextual compatibility across adjacent draft positions. DCOCO maintains parallel drafting while enabling contextually informed token selection at low additional cost. Across multiple verifier backbones and benchmarks, DCOCO achieves the highest average throughput, delivering speedups of up to 6.6× over vanilla decoding and outperforming competing methods by 6–23% under concurrent serving loads.
arXiv Computation and Language
1d ago
HyFlash: A Prefix-Memory Prior for On-Device Speculative Decoding
HyFlash is a compact parallel drafter designed for on-device speculative decoding. It introduces a prefix‑memory prior that separates long‑range context modeling from fine‑grained draft‑position interactions, using local attention for recent context and a recurrent memory to compress historical target features into fixed‑size states. Experiments demonstrate that HyFlash achieves higher acceptance rates than full‑context attention while requiring substantially less KV‑cache storage, especially benefiting shallow drafters and long contexts.
arXiv AI
Sep 30
LongSpark: Efficient speculative decoding with a fixed-cost parallel drafter
arXiv:2609.37029v1 Announce Type: cross Abstract: Speculative decoding accelerates autoregressive inference by verifying multiple draft tokens in a single target forward pass. However, as the context...
arXiv Machine Learning
Sep 1
Strong Drafts Need Compact Memories: Long-Context Speculative Decoding with Compressed KV Cache
arXiv:2608.30252v1 Announce Type: new Abstract: Long-context LLM applications such as document summarization and multi-turn agents require generation from prefixes spanning tens of thousands of token...
Hugging Face Trending Papers
Aug 31
Strong Drafts Need Compact Memories: Long-Context Speculative Decoding with Compressed KV Cache
Long-context LLM applications such as document summarization and multi-turn agents require generation from prefixes spanning tens of thousands of tokens, making decoding latency a major bottleneck. Sp...
arXiv AI
Jun 16
Nightjar: Dynamic Adaptive Speculative Decoding for Large Language Models Serving
arXiv:2512. 22420v5 Announce Type: replace-cross Abstract: Speculative decoding (SD) accelerates LLM inference by verifying draft tokens in parallel.