The paper introduces AdaptiveSpec, a training‑free speculative decoding method that simultaneously adapts the per‑step verification rule and the draft‑tree shape using signals generated during decoding. It replaces the fixed token‑match rule with a margin‑based threshold and adjusts tree depth, width, and node count based on draft confidence and recent acceptance history, allowing the total draft count to vary. Experiments on SGLang show up to 56% throughput gains over EAGLE‑3 while maintaining 93% of lossless task accuracy on GSM8K, MATH‑500, and HumanEval across three models.
arXiv:2604. 02047v2 Announce Type: replace-cross Abstract: Speculative decoding accelerates large language model inference by drafting multiple candidate tokens and verifying them in a single forward pass.
Verification-Aware Training (VAT) is a plug‑in framework that improves speculative decoding for large language models by simulating verification during training and using the resulting accept/reject patterns as supervision. VAT adds a lightweight binary verification head to predict whether each draft token will survive sequential verification, and replaces the fixed per‑position weighting with a verification‑adaptive schedule that keeps full weight up to the first rejection point. When applied to EAGLE‑3 and DFlash on Qwen3‑4B, Qwen3‑8B, and LLaMA‑3.1‑8B, VAT increases average acceptance length by up to 11.4% and wall‑clock speedup by up to 8.7%, yielding consistent gains across math, code, and chat benchmarks.
arXiv:2608.29748v1 Announce Type: new Abstract: Speculative decoding accelerates autoregressive language model inference by having a lightweight draft model propose multiple candidate tokens, which a...
arXiv:2608.20375v1 Announce Type: new Abstract: Tree-based speculative decoding raises the mean accepted tokens of standard speculative decoding by verifying multiple draft paths, and existing tree b...
arXiv:2606. 18967v1 Announce Type: new Abstract: Reinforcement learning (RL) has become a representative post-training paradigm for LLMs, enabling strong reasoning and agentic capabilities.
Osprey is a target‑agnostic pre‑training method that bootstraps draft models for speculative decoding from existing small language models. By pruning to a shallow backbone, restoring language‑modeling capability with next‑token pretraining, and adapting via vocabulary alignment and distillation, Osprey reduces per‑target work to a lightweight adaptation step. Experiments show that a single Osprey backbone improves mean acceptance length by up to 22.7% and increases tokens per second by 17.5% across several large target models, especially on out‑of‑domain and multilingual data.
TreeGraft introduces a multi-drafter framework that combines drafters of varying costs to build a shared draft tree for tree-based speculative decoding. The stronger drafter rescues and rescoring candidates from the weaker drafter, while a lightweight scheduler decides when to invoke the stronger drafter to manage cost. Experiments on 10 model pairs and 6 benchmarks show TreeGraft improves over the best single-drafter strategy by an average of 15.1% and up to 26.6%.
arXiv:2607. 08690v1 Announce Type: cross Abstract: Speculative decoding accelerates sampling from an autoregressive LLM by using a faster auxiliary model to draft tokens which are then verified in parallel by the LLM.
arXiv:2608. 10362v1 Announce Type: cross Abstract: Speculative decoding accelerates autoregressive large language model (LLM) inference by using a lightweight draft model to speculate multiple tokens, reducing expensive target model decoding steps.
arXiv:2608. 03447v1 Announce Type: cross Abstract: Speculative decoding accelerates autoregressive generation by verifying a draft block with a target model in parallel.
arXiv:2606. 03819v1 Announce Type: new Abstract: One-shot block drafters for speculative decoding generate the full draft in a single forward pass, achieving strong throughput by eliminating sequential token generation.
arXiv:2606. 01019v1 Announce Type: cross Abstract: Large Language Model (LLM) generation remains expensive because autoregressive decoding calls the model once for each new token.