arXiv Machine Learning August 4, 2026 By Li Wang, Yi Su, Xiabao Wu, Chiran You, Yongchao Liu, Zhan Qiu, Juelu Zhang, Jiajun Zheng, Fangxin Liu, Jie Zhang, Chen Tian, Chengying Huan

Bole: Efficient Tree Speculation for Hybrid-Attention Language Models

Read the original on arXiv Machine Learning →

arXiv:2608. 01651v1 Announce Type: cross Abstract: Hybrid-attention large language models combine full attention with recurrent linear attention to reduce long-context inference costs, yet their autoregressive decoding remains memory-bound.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Computation and Language

Sep 22

Adapting Tree-Structured Speculative Decoding to DeepSeek-V4 for Efficient Inference

arXiv:2609.24698v1 Announce Type: new Abstract: Repeated execution of the target model during autoregressive decoding is a major source of LLM inference latency. Unlike linear speculation, which foll...

By Changxu Liu, Zhaogeng Li

llms efficiency

arXiv AI

Jun 16

Nightjar: Dynamic Adaptive Speculative Decoding for Large Language Models Serving

arXiv:2512. 22420v5 Announce Type: replace-cross Abstract: Speculative decoding (SD) accelerates LLM inference by verifying draft tokens in parallel.

By Rui Li, Zhaoning Zhang, Libo Zhang, Huaimin Wang, Xiang Fu, Zhiquan Lai

llms efficiency benchmarks

arXiv Machine Learning

Jun 2

Vegas: Self-Speculative Decoding with Verification-Guided Sparse Attention

arXiv:2602. 07223v2 Announce Type: replace Abstract: Long-context large language model (LLM) inference has become the norm for today's AI applications.

By Yikang Yue, Yuqi Xue, Jian Huang

llms efficiency benchmarks

arXiv AI

Jul 1

SMART: When is it Actually Worth Expanding a Speculative Tree?

arXiv:2604. 09731v2 Announce Type: replace-cross Abstract: Tree-based speculative decoding accelerates autoregressive generation by verifying a branching tree of draft tokens in a single target-model forward pass.

By Lifu Wang, Pan Zhou

llms benchmarks

arXiv Machine Learning

Sep 22

H-Spec: Parallel Speculative Decoding Without a Drafter-Side KV Cache

arXiv:2609.24197v1 Announce Type: new Abstract: Speculative decoding losslessly accelerates large language model inference by having a lightweight draft model predict future tokens for verification b...

By Weifan Jiang, Krishna Teja Chitty-Venkata, Megan Flynn, Reed Meyerson, Zhenting Qi, Tianyu Wu, Eldar Kurtic, Minlan Yu, Alexandre Marques

llms diffusion efficiency

arXiv Machine Learning

Sep 21

Elastic Threshold Attention: Learned Contextual Sparsity for Long-Context Decoding

Elastic Threshold Attention (ETA) is a trainable attention mechanism that dynamically predicts contextual thresholds from query representations, enabling selective pruning of KV cache tokens during long‑context decoding. By multiplicatively suppressing sub‑threshold logits during training, ETA avoids representation collapse and eliminates localized attention sinks, allowing a 1.45B model to match dense attention performance at roughly 85% training sparsity and 38% active decode density. At inference, a custom Triton kernel achieves up to 2.5× faster decoding on sequences up to 512K tokens, and an offline calibration step can further reduce compute by 27% by freezing per‑head thresholds.

By Themistoklis Haris, Henry Li, Maryam Karimzadehgan

efficiency

One email a morning, machine-written

Related stories

Adapting Tree-Structured Speculative Decoding to DeepSeek-V4 for Efficient Inference

Nightjar: Dynamic Adaptive Speculative Decoding for Large Language Models Serving

Vegas: Self-Speculative Decoding with Verification-Guided Sparse Attention

SMART: When is it Actually Worth Expanding a Speculative Tree?

H-Spec: Parallel Speculative Decoding Without a Drafter-Side KV Cache

Elastic Threshold Attention: Learned Contextual Sparsity for Long-Context Decoding