arXiv Machine Learning

REAL Loss: Optimizing Real Expected Acceptance Length for Speculative Decoding

The paper introduces REAL, a new training loss that directly optimizes the real expected acceptance length (EAL) for speculative decoding in large language models. Unlike previous block‑level surrogates, REAL uses path weights for unbiased EAL estimation and Rao‑Blackwellization to reduce variance. Experiments with a Qwen3.6‑35B target and Domino drafter show that REAL outperforms existing objectives across multiple tasks and seeds, without altering the drafter architecture or inference process.

arXiv Computation and Language
Nov 4

TVP: TV is Almost All You Need for Speculative Decoding

The paper introduces TVP, a loss family that improves speculative decoding for large language models by focusing on expected acceptance length. It shows that TV, the total variation distance, can match or surpass traditional draft-training objectives when properly tuned, and that TVP’s power parameter balances coverage and acceptance. Experiments across multiple drafter architectures and target models demonstrate that TVP yields higher acceptance lengths, outperforming cross‑entropy and existing training recipes by significant margins.

By Aaryam Sharma
arXiv Machine Learning
Jun 2

LK Losses: Direct Acceptance Rate Optimization for Speculative Decoding

arXiv:2602. 23881v2 Announce Type: replace Abstract: Speculative decoding accelerates autoregressive large language model (LLM) inference by using a lightweight draft model to propose candidate tokens that are then verified in parallel by the target model.

By Alexander Samarin, Sergei Krutikov, Anton Shevtsov, Sergei Skvortsov, Filipp Fisin, Alexander Golubev
arXiv Machine Learning
Oct 8

Training Parallel Speculative Draft Models by Directly Minimizing Expected Decoding Rounds

The paper introduces the Expected Decoding Rounds (EDR) objective, a theoretical framework that models speculative decoding as a Markov reward process and directly optimizes the number of decoding rounds. EDR weights local rejection costs by state occupancies, eliminating auxiliary hyperparameters and enabling unbiased stochastic optimization via temporal‑difference gradients. Applying EDR to fine‑tune state‑of‑the‑art draft models (DSpark and DFly) improves mean accepted length and outperforms prior training objectives across nine benchmarks in math reasoning, code generation, and chat.

By Yunxiao Zhao, Changxiao Cai
arXiv Computation and Language
Oct 19

Acceptance Is Not Speed: Length Inflation in Training-Free Lossy Speculative Decoding

The paper investigates speculative decoding, a technique that speeds up language model inference by generating draft tokens and verifying them. It shows that when verification is relaxed (lossy), the resulting completions become longer—especially due to extended reasoning traces—despite accepting more tokens per round. The study evaluates various relaxed verification rules across multiple model pairs, finding that higher acceptance rates often lead to longer outputs, lower target-model probability tokens, and reduced accuracy, indicating that token acceptance alone is an unreliable metric for decoding speed.

By Chiatzen Wang, Bill Xu, Haochen Gu, Hongyang Zhang
arXiv Computation and Language
Sep 4

Margins, Not Windows: Training-Free Per-Step Lossy Speculative Decoding

The paper introduces AdaptiveSpec, a training‑free speculative decoding method that simultaneously adapts the per‑step verification rule and the draft‑tree shape using signals generated during decoding. It replaces the fixed token‑match rule with a margin‑based threshold and adjusts tree depth, width, and node count based on draft confidence and recent acceptance history, allowing the total draft count to vary. Experiments on SGLang show up to 56% throughput gains over EAGLE‑3 while maintaining 93% of lossless task accuracy on GSM8K, MATH‑500, and HumanEval across three models.

By Oszk\'ar Urb\'an, Young D. Kwon, Stylianos I. Venieris, Cecilia Mascolo
Hugging Face Trending Papers
Jun 10

Breaking Entropy Bounds: Accelerating RL Training via MTP with Rejection Sampling

Reinforcement learning (RL) has become a key component in modern large language models, yet the rollout stage remains the key bottleneck in RL training pipelines. Although Multi-Token Prediction (MTP) offers a natural solution to accelerate rollouts through speculative decoding, many studies have observed that MTP acceptance rates degrade significantly during RL training, leading to limited speedup performance.

arXiv Machine Learning
Jun 11

Breaking Entropy Bounds: Accelerating RL Training via MTP with Rejection Sampling

arXiv:2606. 12370v1 Announce Type: new Abstract: Reinforcement learning (RL) has become a key component in modern large language models, yet the rollout stage remains the key bottleneck in RL training pipelines.

By Yucheng Li, Huiqiang Jiang, Yang Xu, Jianxin Yang, Yi Zhang, Yizhong Cao, Yuhao Shen, Fan Zhou, Rui Men, Jianwei Zhang, An Yang, Bowen Yu, Bo Zheng, Fei Huang, Junyang Lin, Dayiheng Liu, Jingren Zhou