The paper introduces SwitchSD, an adaptive framework that treats copying as a latent control signal in large language model decoding. By training lightweight probes on internal representations, SwitchSD accurately detects genuine copy intent (AUC > 0.99) and dynamically switches between neural drafting and context-based copying. Experiments on Llama and Qwen models show up to 15 % throughput gains over state‑of‑the‑art baselines such as EAGLE3.
arXiv:2607. 10661v1 Announce Type: cross Abstract: Speculative decoding has significantly accelerated Large Language Model (LLM) inference by alleviating memory-bound bottlenecks.
By Zipeng Gao, Zhi Zheng, Qingrong Xia, Junda Lin, Ziwei Zhao, Tong Xu, Zhefeng Wang, Enhong Chen
arXiv:2609.36590v1 Announce Type: cross
Abstract: Self-speculative decoding accelerates large language model (LLM) inference by drafting tokens from the target model itself, but faces a sharp tradeof...
By Hankun Lin, Patrick Pynadath, Ruqi Zhang
Speculative decoding has significantly accelerated Large Language Model (LLM) inference by alleviating memory-bound bottlenecks. However, traditional speculative decoding typically relies on auxiliary draft modules, incurring significant training and communication overhead.
Speculative decoding accelerates generation by verifying multiple drafted tokens in a single target-model forward pass, reducing sequential decoding iterations. Model-free variants avoid auxiliary draft models by reusing text and model states already available during generation, but their speedup depends on the reliability of the constructed drafts.
arXiv:2411. 05894v3 Announce Type: replace-cross Abstract: Speculative Decoding has emerged as a popular technique for accelerating inference in Large Language Models.
By Michele Marzollo, Jiawei Zhuang, Niklas Roemer, Niklas Zwingenberger, Lorenz K. M\"uller, Lukas Cavigelli