arXiv Computation and Language

TreeSpark: Calibrated, Load-Adaptive Draft Trees for Semi-Autoregressive Speculative Decoding

arXiv AI
3d ago

DForkTree: Accelerating Speculative Decoding with Fork-Aware Block Diffusion Draft Trees

DForkTree is a training‑free method that improves speculative decoding by re‑anchoring each branch of a draft tree on its own tokens. It adjusts drafter scores using pointwise mutual information from target‑generated n‑grams, weighting these adjustments more heavily after a fork, and employs a layer‑wise GPU search to recover the exact optimal tree. On Qwen3.5‑4B and 9B models, DForkTree outperforms all baselines, achieving up to 41.5% better results than DFlash, 15.1% better than DDTree, and average speedups of up to 4.75× over autoregressive decoding.

By Zhicheng Zheng, Cheng Zhang, Chenkai Tang, Tianxiang Zhao
arXiv Computation and Language
Aug 28

TreeGraft: Adaptive Multi-Drafter Grafting for Tree-Based Speculative Decoding

TreeGraft introduces a multi-drafter framework that combines drafters of varying costs to build a shared draft tree for tree-based speculative decoding. The stronger drafter rescues and rescoring candidates from the weaker drafter, while a lightweight scheduler decides when to invoke the stronger drafter to manage cost. Experiments on 10 model pairs and 6 benchmarks show TreeGraft improves over the best single-drafter strategy by an average of 15.1% and up to 26.6%.

By Jiaming Fan, Daming Cao, Canchen Huang, Jiale Fu, Jin Zhang, Junjie Gao, Kai Yang, Xiangzhong Luo, Xu Yang
arXiv Computation and Language
Oct 15

TreeSpark: Learning to Draft the Trees You Verify for Speculative Decoding

TreeSpark is a new training and inference framework that extends the semi‑autoregressive DSpark architecture to improve speculative decoding. It builds parent‑conditioned candidate trees in a single backbone pass and then replays their expansion, pruning, and packing during training, using survival negative log‑likelihood and a capped coverage hinge to weight retained edges and promote target‑important alternatives. Evaluations on Qwen3‑4B and Qwen3‑8B across nine general benchmarks and six writing domains show that TreeSpark increases mean acceptance length by 23.4%–31.6% and mean single‑request throughput by 15.1%–25.0% compared to DSpark, with additional gains when using the proposed supervision. whyItMatters":"TreeSpark demonstrates that a unified training and inference approach can substantially boost both acceptance length and throughput in speculative decoding, offering measurable performance improvements for large language models."

By Jihao Zhao, Qianyu Jiang, Shuaishuai Zu, Pengfei Li, Guanghuan Fang, Ben Feng, Biao Qin, Zhibin Wang, Baochuan Yang
arXiv AI
Oct 9

Where Draft Trees Lose Target Mass: Exit-Guided Speculative Decoding

The paper introduces Tree Exit Verification (TEV), an exact, level‑parallel speculative decoding method that uses a single exit‑node decision and a bonus‑token decision to verify draft continuations. By deriving a canonical exit law, the authors show that TEV achieves the same acceptance as optimal verifiers while reducing verifier‑stage latency by 15%. They also propose Exit‑Guided Draft‑Tree Training (ExitTrain), which improves draft trees and increases average output‑block length by 13%, leading to a 14% end‑to‑end speedup over DDTree across dialogue, code, and mathematical reasoning tasks.

By Shijing Hu, Xuancheng Ren, Zhihui Lu, Pan Zhou
arXiv Machine Learning
Oct 3

CAST: Cost-Aware Speculative Trees from One-Pass Block Drafters

CAST (Cost‑Aware Speculative Trees) is a method that improves speculative decoding for large language models by packing multiple drafted token candidates into a tree and verifying the entire tree in a single target‑model pass, rather than only the top‑scoring chain. The tree width is adaptively chosen based on a latency measurement, ensuring that each added candidate’s expected gain outweighs its verification cost. Experiments across five domains, three GPU generations, and two model families show that CAST can be up to 43 % faster than the standard chain, while preserving the target model’s output distribution under both greedy and sampled decoding.

By Jungseob Lee, Sugyeong Eo
Hugging Face Trending Papers
Aug 13

DARTree: Speculative Diffusion Decoding with Autoregressive Draft Trees

Speculative decoding losslessly accelerates autoregressive language models by verifying multiple draft tokens in parallel. Diffusion-based drafters further reduce proposal latency by predicting an entire token block in parallel, but their position-wise distributions are marginal rather than conditioned on tokens selected along each draft path.

arXiv Computation and Language
Oct 7

DLoop: Looped Speculative Decoding

The paper introduces DLoop, a looped speculative decoding technique that adaptively performs multiple drafting stages before verification, allowing a draft model to continue generating tokens while confident. By verifying all accumulated draft tokens together and training the draft model to handle its own hidden states for unverified tokens, DLoop reduces the number of target‑model forward passes needed. Experiments across several speculative decoding methods show wall‑clock speedups of 5–41 % without sacrificing lossless decoding.

By Geonmo Gu, Byeongho Heo, HeeJae Jun, Yoohoon Kang, Sangmin Lee, Sangdoo Yun, Dongyoon Han
arXiv Machine Learning
Oct 28

Bastion: Budget-Aware Speculative Decoding with Tree-structured Block Diffusion Drafting

Bastion is a budget‑aware speculative decoding framework that uses tree‑structured block diffusion drafting to improve the efficiency of language model inference. It dynamically builds query‑dependent trees, balancing draft quality with hardware constraints, and incorporates an acceptance surrogate, an online latency estimator, and an adaptive best‑first expansion to maximize speed without sacrificing model distribution. The method achieves up to a 6.61× speedup over standard autoregressive decoding and outperforms state‑of‑the‑art block‑diffusion baselines by 39% across various benchmarks and GPU architectures.

By Soowon Oh, Nam Cao, Yujin Kim, Hojung Jung, Huzama Ahmad, Sangmin Bae, Se-Young Yun
arXiv AI
Nov 5

Hybrid Spark: Adaptive Speculative Decoding That Lets the Content Choose the Drafter

Hybrid Spark is a framework that runs both a model-based and a model-free speculative decoding drafter on every step, selecting the better chain for each request within adaptive verification entirely on the GPU. By assigning a price per verified draft token based on the goodput that slot would earn elsewhere, it aligns the confidence estimates of both drafters, enabling a single deployment to serve diverse workloads. In experiments, Hybrid Spark achieves 1.10–1.62× higher output throughput than the baseline on code editing tasks while maintaining near‑baseline performance on chat tasks.

By Ekagra Ranjan, Bharat Venkitesh