← Back to all news
arXiv AI By Ekagra Ranjan, Bharat Venkitesh

Hybrid Spark: Adaptive Speculative Decoding That Lets the Content Choose the Drafter

Read the original on arXiv AI →

Hybrid Spark is a framework that runs both a model-based and a model-free speculative decoding drafter on every step, selecting the better chain for each request within adaptive verification entirely on the GPU. By assigning a price per verified draft token based on the goodput that slot would earn elsewhere, it aligns the confidence estimates of both drafters, enabling a single deployment to serve diverse workloads. In experiments, Hybrid Spark achieves 1.10–1.62× higher output throughput than the baseline on code editing tasks while maintaining near‑baseline performance on chat tasks.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
6d ago

Only Accepted Tokens Count: An Energy Accounting of Speculative Decoding Under Latency Targets

The paper investigates how speculative decoding—where a draft model proposes multiple future tokens that are then verified by the target model—affects energy consumption in large language model serving under latency constraints. It finds that energy savings depend heavily on the acceptance rate of drafted tokens, which varies by model architecture, drafting scheme, and the type of text (e.g., conversation vs. code, English vs. Romansh). The study shows that while speculation can reduce energy per token at small batch sizes, at larger batches the verification cost dominates, potentially turning speculation into an energy loss, especially when acceptance rates are low.

By Bole Ma, Ayesha Afzal, Jan Eitzinger, Harald Koestler, Gerhard Wellein
llms
More like this →
arXiv Computation and Language
Oct 7

DLoop: Looped Speculative Decoding

The paper introduces DLoop, a looped speculative decoding technique that adaptively performs multiple drafting stages before verification, allowing a draft model to continue generating tokens while confident. By verifying all accumulated draft tokens together and training the draft model to handle its own hidden states for unverified tokens, DLoop reduces the number of target‑model forward passes needed. Experiments across several speculative decoding methods show wall‑clock speedups of 5–41 % without sacrificing lossless decoding.

By Geonmo Gu, Byeongho Heo, HeeJae Jun, Yoohoon Kang, Sangmin Lee, Sangdoo Yun, Dongyoon Han
llms
More like this →
arXiv AI
Jul 7

DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation

arXiv:2607. 05147v1 Announce Type: new Abstract: Speculative decoding accelerates Large Language Model (LLM) inference by decoupling draft generation from target verification.

By Xin Cheng, Xingkai Yu, Chenze Shao, Jiashi Li, Yunfan Xiong, Yi Qian, Jiaqi Zhu, Shirong Ma, Xiaokang Zhang, Jiasheng Ye, Qinyu Chen, Chengqi Deng, Jiping Yu, Damai Dai, Zhengyan Zhang, Yixuan Wei, Yixuan Tan, Wenkai Yang, Runxin Xu, Yu Wu, Zhean Xu, Xuanyu Wang, Muyang Chen, Rui Tian, Xiao Bi, Zhewen Hao, Shaoyuan Chen, Huanqi Cao, Wentao Zhang, Anyi Xu, Huishuai Zhang, Dongyan Zhao, Wenfeng Liang
llmsefficiencybenchmarks
More like this →