arXiv Machine Learning
FOLD: Position-Free Far Memory with Local Order for Long-Context Speculative Decoding
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Sep 1
Strong Drafts Need Compact Memories: Long-Context Speculative Decoding with Compressed KV Cache
arXiv:2608.30252v1 Announce Type: new Abstract: Long-context LLM applications such as document summarization and multi-turn agents require generation from prefixes spanning tens of thousands of token...
arXiv AI
Sep 30
LongSpark: Efficient speculative decoding with a fixed-cost parallel drafter
arXiv:2609.37029v1 Announce Type: cross Abstract: Speculative decoding accelerates autoregressive inference by verifying multiple draft tokens in a single target forward pass. However, as the context...
Hugging Face Trending Papers
Aug 31
Strong Drafts Need Compact Memories: Long-Context Speculative Decoding with Compressed KV Cache
Long-context LLM applications such as document summarization and multi-turn agents require generation from prefixes spanning tens of thousands of tokens, making decoding latency a major bottleneck. Sp...
arXiv Machine Learning
Jun 2
Vegas: Self-Speculative Decoding with Verification-Guided Sparse Attention
arXiv:2602. 07223v2 Announce Type: replace Abstract: Long-context large language model (LLM) inference has become the norm for today's AI applications.
arXiv Machine Learning
Sep 22
H-Spec: Parallel Speculative Decoding Without a Drafter-Side KV Cache
arXiv:2609.24197v1 Announce Type: new Abstract: Speculative decoding losslessly accelerates large language model inference by having a lightweight draft model predict future tokens for verification b...
arXiv Machine Learning
Jun 25
Dustin: Draft-Augmented Sparse Verification for Efficient Long-Context Generation with Speculative Decoding
arXiv:2606. 24957v1 Announce Type: cross Abstract: While speculative decoding improves inference throughput for multi-batch long-context Large Language Models (LLMs), its efficiency is often limited by a verification bottleneck where Key-Value (KV) cache loading dominates latency.