← Back to all news
arXiv Computation and Language By Chiatzen Wang, Bill Xu, Haochen Gu, Hongyang Zhang

Acceptance Is Not Speed: Length Inflation in Training-Free Lossy Speculative Decoding

Read the original on arXiv Computation and Language →

The paper investigates speculative decoding, a technique that speeds up language model inference by generating draft tokens and verifying them. It shows that when verification is relaxed (lossy), the resulting completions become longer—especially due to extended reasoning traces—despite accepting more tokens per round. The study evaluates various relaxed verification rules across multiple model pairs, finding that higher acceptance rates often lead to longer outputs, lower target-model probability tokens, and reduced accuracy, indicating that token acceptance alone is an unreliable metric for decoding speed.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv Machine Learning
4d ago

Long-Context Drafting Does Not Require Long-Context Drafters

The paper shows that long‑context drafting can be performed with short‑context drafters that are frozen and run beyond their training length, using two inference‑time mechanisms to keep their cost bounded. By bounding the attention cache and re‑basing rotary position indices, the authors demonstrate that the speedup achieved with a short‑context drafter persists up to two million tokens across multiple target‑drafter pairs and corpora. They prove that when a common index shift leaves the drafter’s computation unchanged, the acceptance gap can be tightly bounded, ensuring consistent performance.

By Aref Jafari, Vansh Bhatia, Matt Qin, Subhajit Dutta Chowdhury, Ben Lee, Vikram Appia, Ali Ghodsi, Mehdi Rezagholizadeh
fine-tuning
More like this →
arXiv Machine Learning
Oct 7

APEX: Speculate smarter, not deeper

The paper introduces APEX, a learned controller that improves speculative decoding for large language models by dynamically selecting the best speculation strategy and adjusting draft depth during generation. APEX uses request‑level routing among EAGLE‑3, n‑gram, and draft‑model speculation, and block‑level depth adaptation based on causal decoding signals and verifier feedback. Integrated into vLLM and tested with Qwen3‑8B, APEX achieves up to 5.24× speedup over autoregressive decoding while reducing wasted tokens by 41% compared to fixed‑depth n‑gram speculation.

By Manvi Jha, Zach Zhang, Zhichao Xu, Linbo Liu, Sai Muralidhar Jayanthi, Vinayak Arannil
llmsefficiency
More like this →
arXiv AI
Aug 5

Approximate Speculative Decoding

arXiv:2608. 03447v1 Announce Type: cross Abstract: Speculative decoding accelerates autoregressive generation by verifying a draft block with a target model in parallel.

By Yuannuo Feng, Zegang Peng, Yuxin Xie, Yubing Ye, Yizhe Chen, Wenshuai Yao, Wenyong Zhou, Wang Kang
fine-tuningefficiency
More like this →