arXiv Machine Learning By Vivien Tran-Thien, Richard Nock

Mentored Decoding: Faster Inference meets Boosting

Read the original on arXiv Machine Learning →

The paper introduces mentored decoding, a formal framework for lossy speculative decoding that can accelerate inference of autoregressive language models while potentially improving output quality. It connects this inference technique to boosting theory and extends it to all f‑divergences, revealing geometric insights for total variation, simple approximations tied to boosting compliance, and a divergence‑independent data structure enabling efficient optimal parameter queries and mentored distribution construction.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Machine Learning
Oct 8

Training Parallel Speculative Draft Models by Directly Minimizing Expected Decoding Rounds

The paper introduces the Expected Decoding Rounds (EDR) objective, a theoretical framework that models speculative decoding as a Markov reward process and directly optimizes the number of decoding rounds. EDR weights local rejection costs by state occupancies, eliminating auxiliary hyperparameters and enabling unbiased stochastic optimization via temporal‑difference gradients. Applying EDR to fine‑tune state‑of‑the‑art draft models (DSpark and DFly) improves mean accepted length and outperforms prior training objectives across nine benchmarks in math reasoning, code generation, and chat.

By Yunxiao Zhao, Changxiao Cai
arXiv Machine Learning
Oct 19

SP$^2$ec: Adaptive Self-Speculative Decoding for Vision-Language Models

SP$^2$ec is a training‑free bandit algorithm that adapts the draft configuration during generation for Vision‑Language Models, exploiting a single‑peak throughput pattern observed in existing self‑speculative methods. It minimizes a novel wall‑time regret metric, theoretically guaranteeing near‑optimal draft selection with negligible exploration cost. Experiments on Qwen2.5‑VL‑7B‑Instruct and LLaVA‑Next‑Mistral‑7B show consistent improvements of up to 11% over prior self‑speculative methods across visual and textual tasks.

By Yuqi Huang, Xingyao Li, Yunlong Hou, Fengzhuo Zhang, Jiachun Pan, Vincent Y. F. Tan