arXiv Computation and Language
SFAD: Speculative Factuality-Aware Decoding
Read the original on arXiv Computation and Language →The Flow has not summarised this story yet — read it at arXiv Computation and Language.
Related stories
arXiv AI
3d ago
Entropy-Aware Token Rejection for Improving Speculative Decoding
arXiv:2512.23765v2 Announce Type: replace-cross Abstract: Speculative decoding (SD) accelerates large language model (LLM) inference by using a lightweight draft model to propose tokens and a stronge...
arXiv Computation and Language
2d ago
Beyond Tokens: Semantic-Aware Speculative Decoding for Efficient Inference by Probing Internal States
Large Language Models suffer from high inference latency, especially when generating long chains of thought. Existing speculative decoding methods draft and verify tokens in parallel but ignore semantic equivalence, causing inefficient rejections. The proposed SemanticSpec framework verifies entire semantic sequences by probing internal hidden states, achieving up to 2.7× speedup on DeepSeekR1-32B and 2.1× on QwQ-32B while outperforming token‑level and sequence‑level baselines in both efficiency and effectiveness.
arXiv Machine Learning
Jul 13
Accelerating Large Language Model Inference with Self-Supervised Early Exits
arXiv:2407. 21082v3 Announce Type: replace-cross Abstract: This paper presents a modular approach to accelerate inference in large language models (LLMs) by adding early exit heads at intermediate transformer layers.
Hugging Face Trending Papers
Jul 29
Revisiting Lossy Verification in Speculative Decoding: Mechanisms, Trade-offs, and Failure Modes
Speculative Decoding (SD) accelerates large language model inference by allowing a lightweight draft model to propose tokens that are subsequently verified in parallel by a larger target model. Recent approaches introduce lossy verification schemes to further improve efficiency by relaxing strict distributional matching.
arXiv AI
Jul 15
Less Experts, Faster Decoding: Cost-Aware Speculative Decoding for Mixture-of-Experts
arXiv:2607. 12696v1 Announce Type: cross Abstract: Sparse Mixture-of-Experts (MoE) models have become an important approach for scaling Large Language Models (LLMs), but their inference efficiency depends strongly on expert activation patterns.
arXiv Machine Learning
Jun 29
EntMTP: Accelerating LLM Inference with Entropy Guided Multi Token Prediction
arXiv:2606. 27550v1 Announce Type: cross Abstract: Multi-token prediction has been shown to increase data density during training, improve downstream text-generation quality, and serves as the defacto approach for self-speculative decoding.