arXiv AI

ForeSpark: Joint Draft and Verification Budget Scheduling for Parallel Speculative Decoding

ForeSpark is a prototype that implements work‑conserving speculation (WCS) to jointly schedule draft and verification budgets for parallel speculative decoding in large language model serving. It uses a context‑aware pre‑draft estimator, a lightweight joint scheduler, and a post‑draft budget distributor to predict and allocate draft lengths and verification effort efficiently. The system, built on vLLM, achieves up to 1.77× higher throughput than DSpark and 2.49× higher than autoregressive decoding while maintaining the same output distribution and supporting higher request rates under latency SLOs.

Hugging Face Trending Papers
Nov 10

ForeSpark: Joint Draft and Verification Budget Scheduling for Parallel Speculative Decoding

ForeSpark introduces a work‑conserving speculation (WCS) framework that jointly schedules draft and verification budgets for parallel speculative decoding in large language model serving. It uses a context‑aware pre‑draft estimator, a lightweight joint scheduler, and a post‑draft budget allocator to reduce wasted compute and improve throughput. Experiments on vLLM show up to 1.77× higher throughput than DSpark and 2.49× over autoregressive decoding, while maintaining the same output distribution and supporting higher request rates under latency SLOs.

arXiv Machine Learning
Oct 7

APEX: Speculate smarter, not deeper

The paper introduces APEX, a learned controller that improves speculative decoding for large language models by dynamically selecting the best speculation strategy and adjusting draft depth during generation. APEX uses request‑level routing among EAGLE‑3, n‑gram, and draft‑model speculation, and block‑level depth adaptation based on causal decoding signals and verifier feedback. Integrated into vLLM and tested with Qwen3‑8B, APEX achieves up to 5.24× speedup over autoregressive decoding while reducing wasted tokens by 41% compared to fixed‑depth n‑gram speculation.

By Manvi Jha, Zach Zhang, Zhichao Xu, Linbo Liu, Sai Muralidhar Jayanthi, Vinayak Arannil
arXiv AI
Jul 7

DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation

arXiv:2607. 05147v1 Announce Type: new Abstract: Speculative decoding accelerates Large Language Model (LLM) inference by decoupling draft generation from target verification.

By Xin Cheng, Xingkai Yu, Chenze Shao, Jiashi Li, Yunfan Xiong, Yi Qian, Jiaqi Zhu, Shirong Ma, Xiaokang Zhang, Jiasheng Ye, Qinyu Chen, Chengqi Deng, Jiping Yu, Damai Dai, Zhengyan Zhang, Yixuan Wei, Yixuan Tan, Wenkai Yang, Runxin Xu, Yu Wu, Zhean Xu, Xuanyu Wang, Muyang Chen, Rui Tian, Xiao Bi, Zhewen Hao, Shaoyuan Chen, Huanqi Cao, Wentao Zhang, Anyi Xu, Huishuai Zhang, Dongyan Zhao, Wenfeng Liang
arXiv AI
Nov 5

Hybrid Spark: Adaptive Speculative Decoding That Lets the Content Choose the Drafter

Hybrid Spark is a framework that runs both a model-based and a model-free speculative decoding drafter on every step, selecting the better chain for each request within adaptive verification entirely on the GPU. By assigning a price per verified draft token based on the goodput that slot would earn elsewhere, it aligns the confidence estimates of both drafters, enabling a single deployment to serve diverse workloads. In experiments, Hybrid Spark achieves 1.10–1.62× higher output throughput than the baseline on code editing tasks while maintaining near‑baseline performance on chat tasks.

By Ekagra Ranjan, Bharat Venkitesh
arXiv Computation and Language
Oct 7

DLoop: Looped Speculative Decoding

The paper introduces DLoop, a looped speculative decoding technique that adaptively performs multiple drafting stages before verification, allowing a draft model to continue generating tokens while confident. By verifying all accumulated draft tokens together and training the draft model to handle its own hidden states for unverified tokens, DLoop reduces the number of target‑model forward passes needed. Experiments across several speculative decoding methods show wall‑clock speedups of 5–41 % without sacrificing lossless decoding.

By Geonmo Gu, Byeongho Heo, HeeJae Jun, Yoohoon Kang, Sangmin Lee, Sangdoo Yun, Dongyoon Han
arXiv Machine Learning
Oct 8

Training Parallel Speculative Draft Models by Directly Minimizing Expected Decoding Rounds

The paper introduces the Expected Decoding Rounds (EDR) objective, a theoretical framework that models speculative decoding as a Markov reward process and directly optimizes the number of decoding rounds. EDR weights local rejection costs by state occupancies, eliminating auxiliary hyperparameters and enabling unbiased stochastic optimization via temporal‑difference gradients. Applying EDR to fine‑tune state‑of‑the‑art draft models (DSpark and DFly) improves mean accepted length and outperforms prior training objectives across nine benchmarks in math reasoning, code generation, and chat.

By Yunxiao Zhao, Changxiao Cai
arXiv Machine Learning
Sep 17

ASPIRE: Asynchronous Batched Self-Speculative Decoding for Long-Context LLM Inference

ASPIRE introduces a non‑synchronized batched self‑speculative decoding framework for long‑context LLM inference, addressing the memory bottleneck of attention by drafting tokens with sparse attention and verifying them with full attention. It combines a unified mixed forward pass, a lightweight online speculation scheduler that lets each request independently decide when to verify, and an intra‑draft refresh layer that updates the sparse context at every draft step. Experiments on three models and five benchmarks show 1.70–4.58× speedup over autoregressive baselines and a 27% average improvement over the strongest prior self‑speculative methods.

By Amir Ziashahabi, Hossein Entezari Zarch, Lei Gao, Murali Annavaram, Salman Avestimehr
arXiv Machine Learning
Oct 16

Long-Context Drafting Does Not Require Long-Context Drafters

The paper shows that long‑context drafting can be performed with short‑context drafters that are frozen and run beyond their training length, using two inference‑time mechanisms to keep their cost bounded. By bounding the attention cache and re‑basing rotary position indices, the authors demonstrate that the speedup achieved with a short‑context drafter persists up to two million tokens across multiple target‑drafter pairs and corpora. They prove that when a common index shift leaves the drafter’s computation unchanged, the acceptance gap can be tightly bounded, ensuring consistent performance.

By Aref Jafari, Vansh Bhatia, Matt Qin, Subhajit Dutta Chowdhury, Ben Lee, Vikram Appia, Ali Ghodsi, Mehdi Rezagholizadeh
arXiv AI
Oct 31

Only Accepted Tokens Count: An Energy Accounting of Speculative Decoding Under Latency Targets

The paper investigates how speculative decoding—where a draft model proposes multiple future tokens that are then verified by the target model—affects energy consumption in large language model serving under latency constraints. It finds that energy savings depend heavily on the acceptance rate of drafted tokens, which varies by model architecture, drafting scheme, and the type of text (e.g., conversation vs. code, English vs. Romansh). The study shows that while speculation can reduce energy per token at small batch sizes, at larger batches the verification cost dominates, potentially turning speculation into an energy loss, especially when acceptance rates are low.

By Bole Ma, Ayesha Afzal, Jan Eitzinger, Harald Koestler, Gerhard Wellein