The paper examines lossy verification techniques used in speculative decoding for large language models, showing that many methods can be grouped into truncation-based and collaborative verification categories. It analyzes how these approaches alter the decoding distribution, revealing that truncation-based methods can significantly degrade performance due to distributional distortion, while collaborative methods depend more on overshoot suppression and supervision quality than on simple interpolation between draft and target models. A diagnostic evaluation framework is introduced to assess these failure modes across curated benchmarks.
By Tianyu Wang, Yuxuan Zhou, Heng Li, Wenbin Wang, Zikai Xiao, Chunrui Zheng, Junyuan Shang
The efficiency of Large Language Model (LLM) serving is fundamentally limited by the sequential nature of autoregressive decoding. Speculative Decoding (SD) mitigates this by using a lightweight draft...
ResiSpec is a framework that improves speculative decoding for large language models by reshaping the residual distribution during verification. It addresses the problem of residual drift, where rejected candidates cause the target distribution to diverge from the draft model’s predictions, rendering later candidates ineffective. By aligning the verification process with the draft model’s high‑confidence regions, ResiSpec prevents candidate obsolescence and achieves up to 1.92× speedup over existing multi‑candidate methods.
By Zhi-Kai Chen, Jun-Jie Tao, Wei-Xiang Mao, De-Chuan Zhan, Han-Jia Ye
arXiv:2607. 08690v1 Announce Type: cross Abstract: Speculative decoding accelerates sampling from an autoregressive LLM by using a faster auxiliary model to draft tokens which are then verified in parallel by the LLM.
By Guoxuan Xia, Luka Ribar, Paul Balanca
arXiv:2608. 08721v1 Announce Type: cross Abstract: Speculative decoding accelerates large language model inference by drafting multiple tokens for parallel verification, with efficiency critically determined by the speculative length selected at each decoding round.
By Zexun Lin, Yuan Feng, Junlin Lv, Kevin S. Zhou, Xike Xie
Speculative decoding accelerates sampling from an autoregressive LLM by using a faster auxiliary model to draft tokens which are then verified in parallel by the LLM. Standard speculative decoding is lossless: its rejection and resampling steps exactly preserve the LLM's sampling distribution.
arXiv:2512.23765v2 Announce Type: replace-cross
Abstract: Speculative decoding (SD) accelerates large language model (LLM) inference by using a lightweight draft model to propose tokens and a stronge...
By Tiancheng Su, Meicong Zhang, Guoxiu He
arXiv:2606. 04446v1 Announce Type: cross Abstract: Speculative decoding accelerates autoregressive large language model inference by drafting multiple tokens and verifying them in a single target-model forward pass.
By Liyuan Zhang, Jiarui Zhang, Jinwei Yao, Ran Yan, Yuchen Yang, Jiahao Zhang, Tongkai Yang, Yi Wu, Binhang Yuan
arXiv:2608.29748v1 Announce Type: new
Abstract: Speculative decoding accelerates autoregressive language model inference by having a lightweight draft model propose multiple candidate tokens, which a...
By Luxi Lin, Zhanpeng Zeng, Shuang Peng, Songwei Liu, Rongrong Ji
arXiv:2512.02337v2 Announce Type: replace
Abstract: Growing demands from tasks like code generation, deep reasoning, and long-document understanding have made long-context generation a crucial capabi...
By Zhendong Tan, Xingjun Zhang, Chaoyi Hu, Junjie Peng, Kun Xia
arXiv:2606. 01019v1 Announce Type: cross Abstract: Large Language Model (LLM) generation remains expensive because autoregressive decoding calls the model once for each new token.
By Xin Su, Dawid Majchrowski, Fangyuan Yu, Vanshil Atul Shah, Sebastian Rogawski, Pawel Morkisz, Anahita Bhiwandiwalla, Phillip Howard
arXiv:2605. 30580v2 Announce Type: replace-cross Abstract: Speculative decoding is a popular technique for large language model (LLM) inference, enabling faster generation by drafting multiple tokens with a smaller draft model.
By Nirajan Paudel, Michael Ginn, Luc De Nardi, Alexis Palmer