arXiv Machine Learning
Strong Drafts Need Compact Memories: Long-Context Speculative Decoding with Compressed KV Cache
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
Hugging Face Trending Papers
Aug 31
Strong Drafts Need Compact Memories: Long-Context Speculative Decoding with Compressed KV Cache
Long-context LLM applications such as document summarization and multi-turn agents require generation from prefixes spanning tens of thousands of tokens, making decoding latency a major bottleneck. Sp...
arXiv Machine Learning
Oct 15
FOLD: Position-Free Far Memory with Local Order for Long-Context Speculative Decoding
arXiv:2610.20392v1 Announce Type: cross Abstract: Speculative decoding accelerates LLM generation through lightweight drafting and parallel verification. It is particularly promising for long inputs,...
arXiv Machine Learning
Sep 22
H-Spec: Parallel Speculative Decoding Without a Drafter-Side KV Cache
arXiv:2609.24197v1 Announce Type: new Abstract: Speculative decoding losslessly accelerates large language model inference by having a lightweight draft model predict future tokens for verification b...
arXiv AI
Sep 30
LongSpark: Efficient speculative decoding with a fixed-cost parallel drafter
arXiv:2609.37029v1 Announce Type: cross Abstract: Speculative decoding accelerates autoregressive inference by verifying multiple draft tokens in a single target forward pass. However, as the context...
arXiv AI
Oct 5
BitNest: Bit-Nested Speculative Decoding for Memory-Efficient LLM Inference Acceleration
arXiv:2610.02800v1 Announce Type: new Abstract: Speculative decoding accelerates autoregressive generation by using a lightweight draft to propose multiple tokens for parallel verification. However,...
arXiv AI
Jun 2
BudgetDraft: Acceptance-Aware Multi-View Training for Sparse-KV Speculative Decoding
arXiv:2606. 00144v1 Announce Type: cross Abstract: Speculative decoding speeds up autoregressive decoding by using a drafter to propose multiple tokens that a verifier validates in parallel.