arXiv AI
Adaptive Multi-Token Prediction: How far should we go?
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Oct 2
Match the Distribution, Not the Compute: Post-Training Multi-Token Prediction Heads
arXiv:2610.00888v1 Announce Type: cross Abstract: Multi-token prediction (MTP) improves the throughput of autoregressive generation by enabling the language model to draft multiple next tokens per fo...
arXiv Computation and Language
Oct 1
DEdit: Iterative Draft Editing for Speculative Decoding
arXiv:2609.38510v1 Announce Type: new Abstract: Speculative decoding accelerates autoregressive LLMs by having a lightweight drafter propose tokens that the target model verifies in parallel. Diffusi...
arXiv Machine Learning
Oct 7
APEX: Speculate smarter, not deeper
The paper introduces APEX, a learned controller that improves speculative decoding for large language models by dynamically selecting the best speculation strategy and adjusting draft depth during generation. APEX uses request‑level routing among EAGLE‑3, n‑gram, and draft‑model speculation, and block‑level depth adaptation based on causal decoding signals and verifier feedback. Integrated into vLLM and tested with Qwen3‑8B, APEX achieves up to 5.24× speedup over autoregressive decoding while reducing wasted tokens by 41% compared to fixed‑depth n‑gram speculation.
arXiv Machine Learning
Oct 16
Long-Context Drafting Does Not Require Long-Context Drafters
The paper shows that long‑context drafting can be performed with short‑context drafters that are frozen and run beyond their training length, using two inference‑time mechanisms to keep their cost bounded. By bounding the attention cache and re‑basing rotary position indices, the authors demonstrate that the speedup achieved with a short‑context drafter persists up to two million tokens across multiple target‑drafter pairs and corpora. They prove that when a common index shift leaves the drafter’s computation unchanged, the acceptance gap can be tightly bounded, ensuring consistent performance.
arXiv Computation and Language
Oct 7
DLoop: Looped Speculative Decoding
The paper introduces DLoop, a looped speculative decoding technique that adaptively performs multiple drafting stages before verification, allowing a draft model to continue generating tokens while confident. By verifying all accumulated draft tokens together and training the draft model to handle its own hidden states for unverified tokens, DLoop reduces the number of target‑model forward passes needed. Experiments across several speculative decoding methods show wall‑clock speedups of 5–41 % without sacrificing lossless decoding.
arXiv Computation and Language
Oct 5
From Positionwise Confidence to Prefix Scheduling: Verifier Skipping in Speculative Decoding
The paper explores a new lossy strategy for speculative decoding called verifier skipping, which allows certain draft prefixes to be committed without invoking the verifier. It investigates which confidence signals—raw token confidence, learned marginal survival, or learned conditional survival—best schedule these skips, finding that raw confidence often performs best despite higher positionwise AUROC for learned signals. Experiments on HumanEval with DiffuCoder-7B-Instruct and Qwen3-32B show that all three signals can reduce verifier calls by 9.6% to 13.5% while maintaining the same pass@1 as Strict SDD.