arXiv Machine Learning By Joseph Rowan, Ashish J. Khisti

Multiple Importance Sampling for Guided Speculative Inference

Read the original on arXiv Machine Learning →

The paper extends Guided Speculative Inference (GSI) for test‑time scaling of large language models by allowing an ensemble of smaller LLMs to propose draft reasoning steps, which are then verified by a larger base model and a reward model. Using multiple importance sampling (MIS), the authors introduce weighting schemes to combine samples from diverse proposal distributions, prove new bounds on KL divergence and expected reward loss, and show that the balance heuristic is optimal. Experiments demonstrate that combining several diverse LLMs, while keeping the total number of samples fixed, achieves competitive downstream performance and is mathematically robust when a single draft model’s support does not cover the target distribution, a scenario where standard GSI may fail.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Machine Learning
Oct 8

Training Parallel Speculative Draft Models by Directly Minimizing Expected Decoding Rounds

The paper introduces the Expected Decoding Rounds (EDR) objective, a theoretical framework that models speculative decoding as a Markov reward process and directly optimizes the number of decoding rounds. EDR weights local rejection costs by state occupancies, eliminating auxiliary hyperparameters and enabling unbiased stochastic optimization via temporal‑difference gradients. Applying EDR to fine‑tune state‑of‑the‑art draft models (DSpark and DFly) improves mean accepted length and outperforms prior training objectives across nine benchmarks in math reasoning, code generation, and chat.

By Yunxiao Zhao, Changxiao Cai
arXiv Computation and Language
Sep 21

Draft-OPD: On-Policy Distillation for Speculative Draft Models

Draft-OPD introduces an on‑policy distillation method for speculative draft models, addressing the mismatch between supervised fine‑tuning and inference by letting the target model supervise the drafter on draft‑induced states. The approach uses target‑assisted rollouts for stable continuations and replays drafting from error positions exposed during verification, enabling the drafter to learn from both accepted and rejected proposals. Experiments demonstrate that Draft‑OPD achieves more than five‑fold lossless acceleration across diverse tasks, outperforming prior draft models such as EAGLE‑3 and DFlash by 23 % and 13 % respectively.

By Haodi Lei, Yafu Li, Haoran Zhang, Shunkai Zhang, Qianjia Cheng, Xiaoye Qu, Ganqu Cui, Bowen Zhou, Ning Ding, Yun Luo, Yu Cheng
arXiv Machine Learning
Nov 5

Give the Drafter a Second Chance to Guess: Parallel Reconsideration for Speculative Decoding

The paper introduces Parallel Reconsideration, a technique that lets a drafter in speculative decoding re-evaluate uncertain token guesses with additional parallel passes, adding minimal latency. This method, combined with a new training recipe and drafting algorithm, enhances the drafter’s acceptance rate and overall speedup. Experiments across models and benchmarks show average improvements of 5.3%–10.2% in acceptance length and 3.5%–9.2% in speedup.

By Hoang Anh Duy Le, Huibing Dong, Alireza Khadem, Ziteng Sun, Xiang Song, Yizheng Huang, Soroush Radpour, Anshumali Shrivastava, Ye He
arXiv Machine Learning
Oct 5

Rank-Aware Speculative Sampling for Diffusion Draft Trees

The paper introduces Rank-Aware Speculative Sampling (RASS), a new verification rule for speculative draft trees in diffusion models. RASS orders draft candidates by proposal‑target mean displacement, samples a rank with weights optimized to minimize total variation, and then maximally couples the selected candidate with the target distribution, ensuring exact sampling. Experiments on Gaussian mixtures, FFHQ, CIFAR‑10, and Stable Diffusion 3.5 show that RASS reduces target‑model evaluation counts by up to 20% compared to Diffusion Greedy Rejection Sampling (D‑GRS).

By Marcello Bullo, Yanxiao Liu, \"Oyk\"u S{\i}la G\"uner, Arpan Mukherjee, Deniz G\"und\"uz