arXiv AI
3d ago

DForkTree: Accelerating Speculative Decoding with Fork-Aware Block Diffusion Draft Trees

DForkTree is a training‑free method that improves speculative decoding by re‑anchoring each branch of a draft tree on its own tokens. It adjusts drafter scores using pointwise mutual information from target‑generated n‑grams, weighting these adjustments more heavily after a fork, and employs a layer‑wise GPU search to recover the exact optimal tree. On Qwen3.5‑4B and 9B models, DForkTree outperforms all baselines, achieving up to 41.5% better results than DFlash, 15.1% better than DDTree, and average speedups of up to 4.75× over autoregressive decoding.

By Zhicheng Zheng, Cheng Zhang, Chenkai Tang, Tianxiang Zhao
arXiv Computation and Language
Aug 28

TreeGraft: Adaptive Multi-Drafter Grafting for Tree-Based Speculative Decoding

TreeGraft introduces a multi-drafter framework that combines drafters of varying costs to build a shared draft tree for tree-based speculative decoding. The stronger drafter rescues and rescoring candidates from the weaker drafter, while a lightweight scheduler decides when to invoke the stronger drafter to manage cost. Experiments on 10 model pairs and 6 benchmarks show TreeGraft improves over the best single-drafter strategy by an average of 15.1% and up to 26.6%.

By Jiaming Fan, Daming Cao, Canchen Huang, Jiale Fu, Jin Zhang, Junjie Gao, Kai Yang, Xiangzhong Luo, Xu Yang
arXiv Computation and Language
Oct 15

TreeSpark: Learning to Draft the Trees You Verify for Speculative Decoding

TreeSpark is a new training and inference framework that extends the semi‑autoregressive DSpark architecture to improve speculative decoding. It builds parent‑conditioned candidate trees in a single backbone pass and then replays their expansion, pruning, and packing during training, using survival negative log‑likelihood and a capped coverage hinge to weight retained edges and promote target‑important alternatives. Evaluations on Qwen3‑4B and Qwen3‑8B across nine general benchmarks and six writing domains show that TreeSpark increases mean acceptance length by 23.4%–31.6% and mean single‑request throughput by 15.1%–25.0% compared to DSpark, with additional gains when using the proposed supervision. whyItMatters":"TreeSpark demonstrates that a unified training and inference approach can substantially boost both acceptance length and throughput in speculative decoding, offering measurable performance improvements for large language models."

By Jihao Zhao, Qianyu Jiang, Shuaishuai Zu, Pengfei Li, Guanghuan Fang, Ben Feng, Biao Qin, Zhibin Wang, Baochuan Yang
arXiv AI
Oct 9

Where Draft Trees Lose Target Mass: Exit-Guided Speculative Decoding

The paper introduces Tree Exit Verification (TEV), an exact, level‑parallel speculative decoding method that uses a single exit‑node decision and a bonus‑token decision to verify draft continuations. By deriving a canonical exit law, the authors show that TEV achieves the same acceptance as optimal verifiers while reducing verifier‑stage latency by 15%. They also propose Exit‑Guided Draft‑Tree Training (ExitTrain), which improves draft trees and increases average output‑block length by 13%, leading to a 14% end‑to‑end speedup over DDTree across dialogue, code, and mathematical reasoning tasks.

By Shijing Hu, Xuancheng Ren, Zhihui Lu, Pan Zhou
arXiv Machine Learning
Oct 3

CAST: Cost-Aware Speculative Trees from One-Pass Block Drafters

CAST (Cost‑Aware Speculative Trees) is a method that improves speculative decoding for large language models by packing multiple drafted token candidates into a tree and verifying the entire tree in a single target‑model pass, rather than only the top‑scoring chain. The tree width is adaptively chosen based on a latency measurement, ensuring that each added candidate’s expected gain outweighs its verification cost. Experiments across five domains, three GPU generations, and two model families show that CAST can be up to 43 % faster than the standard chain, while preserving the target model’s output distribution under both greedy and sampled decoding.

By Jungseob Lee, Sugyeong Eo