arXiv Machine Learning

Mentored Decoding: Faster Inference meets Boosting

The paper introduces mentored decoding, a formal framework for lossy speculative decoding that can accelerate inference of autoregressive language models while potentially improving output quality. It connects this inference technique to boosting theory and extends it to all f‑divergences, revealing geometric insights for total variation, simple approximations tied to boosting compliance, and a divergence‑independent data structure enabling efficient optimal parameter queries and mentored distribution construction.

arXiv Machine Learning
Oct 8

Training Parallel Speculative Draft Models by Directly Minimizing Expected Decoding Rounds

The paper introduces the Expected Decoding Rounds (EDR) objective, a theoretical framework that models speculative decoding as a Markov reward process and directly optimizes the number of decoding rounds. EDR weights local rejection costs by state occupancies, eliminating auxiliary hyperparameters and enabling unbiased stochastic optimization via temporal‑difference gradients. Applying EDR to fine‑tune state‑of‑the‑art draft models (DSpark and DFly) improves mean accepted length and outperforms prior training objectives across nine benchmarks in math reasoning, code generation, and chat.

By Yunxiao Zhao, Changxiao Cai
arXiv Machine Learning
Oct 19

SP$^2$ec: Adaptive Self-Speculative Decoding for Vision-Language Models

SP$^2$ec is a training‑free bandit algorithm that adapts the draft configuration during generation for Vision‑Language Models, exploiting a single‑peak throughput pattern observed in existing self‑speculative methods. It minimizes a novel wall‑time regret metric, theoretically guaranteeing near‑optimal draft selection with negligible exploration cost. Experiments on Qwen2.5‑VL‑7B‑Instruct and LLaVA‑Next‑Mistral‑7B show consistent improvements of up to 11% over prior self‑speculative methods across visual and textual tasks.

By Yuqi Huang, Xingyao Li, Yunlong Hou, Fengzhuo Zhang, Jiachun Pan, Vincent Y. F. Tan
arXiv AI
Aug 26

ResiSpec: Enhancing Multi-Candidate Speculative Sampling via Residual Distribution Shaping

ResiSpec is a framework that improves speculative decoding for large language models by reshaping the residual distribution during verification. It addresses the problem of residual drift, where rejected candidates cause the target distribution to diverge from the draft model’s predictions, rendering later candidates ineffective. By aligning the verification process with the draft model’s high‑confidence regions, ResiSpec prevents candidate obsolescence and achieves up to 1.92× speedup over existing multi‑candidate methods.

By Zhi-Kai Chen, Jun-Jie Tao, Wei-Xiang Mao, De-Chuan Zhan, Han-Jia Ye
Hugging Face Trending Papers
Jun 10

Teaching Diffusion to Speculate Left-to-Right

Large language models (LLMs) achieve remarkable performance across a wide range of tasks, but their autoregressive decoding process incurs substantial inference costs due to inherently sequential token generation. Speculative decoding addresses this bottleneck by employing a lightweight draft model to propose multiple future tokens that are subsequently verified in parallel by a larger target model.

arXiv Machine Learning
6d ago

REAL Loss: Optimizing Real Expected Acceptance Length for Speculative Decoding

The paper introduces REAL, a new training loss that directly optimizes the real expected acceptance length (EAL) for speculative decoding in large language models. Unlike previous block‑level surrogates, REAL uses path weights for unbiased EAL estimation and Rao‑Blackwellization to reduce variance. Experiments with a Qwen3.6‑35B target and Domino drafter show that REAL outperforms existing objectives across multiple tasks and seeds, without altering the drafter architecture or inference process.

By Yu Tian