arXiv Machine Learning
Twig: Parallel and Memory-Efficient Tree Speculative Decoding for Hybrid-Attention LLMs
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Aug 4
Bole: Efficient Tree Speculation for Hybrid-Attention Language Models
arXiv:2608. 01651v1 Announce Type: cross Abstract: Hybrid-attention large language models combine full attention with recurrent linear attention to reduce long-context inference costs, yet their autoregressive decoding remains memory-bound.
arXiv AI
Jun 16
Nightjar: Dynamic Adaptive Speculative Decoding for Large Language Models Serving
arXiv:2512. 22420v5 Announce Type: replace-cross Abstract: Speculative decoding (SD) accelerates LLM inference by verifying draft tokens in parallel.
arXiv Computation and Language
Sep 22
Adapting Tree-Structured Speculative Decoding to DeepSeek-V4 for Efficient Inference
arXiv:2609.24698v1 Announce Type: new Abstract: Repeated execution of the target model during autoregressive decoding is a major source of LLM inference latency. Unlike linear speculation, which foll...
arXiv AI
Jul 1
SMART: When is it Actually Worth Expanding a Speculative Tree?
arXiv:2604. 09731v2 Announce Type: replace-cross Abstract: Tree-based speculative decoding accelerates autoregressive generation by verifying a branching tree of draft tokens in a single target-model forward pass.
arXiv AI
Jul 24
SonicSampler: Unified Tile-Aware Kernels for LLM Sampling and Speculative Verification
arXiv:2607. 20475v1 Announce Type: new Abstract: Sampling in LLM inference comprises a combinatorial set of logit processing, token selection, and verification operations for speculative decoding.
arXiv Machine Learning
Oct 28
Bastion: Budget-Aware Speculative Decoding with Tree-structured Block Diffusion Drafting
Bastion is a budget‑aware speculative decoding framework that uses tree‑structured block diffusion drafting to improve the efficiency of language model inference. It dynamically builds query‑dependent trees, balancing draft quality with hardware constraints, and incorporates an acceptance surrogate, an online latency estimator, and an adaptive best‑first expansion to maximize speed without sacrificing model distribution. The method achieves up to a 6.61× speedup over standard autoregressive decoding and outperforms state‑of‑the‑art block‑diffusion baselines by 39% across various benchmarks and GPU architectures.