arXiv:2407. 21082v3 Announce Type: replace-cross Abstract: This paper presents a modular approach to accelerate inference in large language models (LLMs) by adding early exit heads at intermediate transformer layers.
arXiv:2609.36590v1 Announce Type: cross Abstract: Self-speculative decoding accelerates large language model (LLM) inference by drafting tokens from the target model itself, but faces a sharp tradeof...
arXiv:2411. 05894v3 Announce Type: replace-cross Abstract: Speculative Decoding has emerged as a popular technique for accelerating inference in Large Language Models.
arXiv:2609.17184v1 Announce Type: new Abstract: Looped Transformers achieve strong performance with compact parameter sizes by repeatedly applying a shared stack of Transformer blocks across recurren...
arXiv:2607. 10661v1 Announce Type: cross Abstract: Speculative decoding has significantly accelerated Large Language Model (LLM) inference by alleviating memory-bound bottlenecks.
Speculative decoding has significantly accelerated Large Language Model (LLM) inference by alleviating memory-bound bottlenecks. However, traditional speculative decoding typically relies on auxiliary draft modules, incurring significant training and communication overhead.
arXiv:2512.23765v2 Announce Type: replace-cross Abstract: Speculative decoding (SD) accelerates large language model (LLM) inference by using a lightweight draft model to propose tokens and a stronge...
arXiv:2605. 30580v2 Announce Type: replace-cross Abstract: Speculative decoding is a popular technique for large language model (LLM) inference, enabling faster generation by drafting multiple tokens with a smaller draft model.
Large Language Models suffer from high inference latency, especially when generating long chains of thought. Existing speculative decoding methods draft and verify tokens in parallel but ignore semantic equivalence, causing inefficient rejections. The proposed SemanticSpec framework verifies entire semantic sequences by probing internal hidden states, achieving up to 2.7× speedup on DeepSeekR1-32B and 2.1× on QwQ-32B while outperforming token‑level and sequence‑level baselines in both efficiency and effectiveness.
FlexEE is an early‑exiting framework designed for large language model inference that is constrained by computation and memory, particularly in offloading‑based deployments. It uses layer‑wise exit supervision, self‑speculative decoding over a Top‑K local vocabulary, and dynamic hidden‑state management to enable reliable intermediate‑layer predictions and memory‑aware execution. Experiments on Llama2‑7B and Llama3‑8B show that FlexEE achieves significant speedups—up to 1.27×/3.16× and 1.25×/2.83× respectively—while maintaining minimal accuracy loss.
arXiv:2606. 04446v1 Announce Type: cross Abstract: Speculative decoding accelerates autoregressive large language model inference by drafting multiple tokens and verifying them in a single target-model forward pass.
The paper introduces Prediction of Prediction (PoP), a method that fuses intermediate hidden representations across transformer layers during a single forward pass to detect hallucinations in large language models. PoP leverages internal hidden‑state transition dynamics to signal factual errors without extra decoding steps, achieving a 75.5% AUROC on the TruthfulQA benchmark with less than 1.2% added latency.
arXiv:2512.02337v2 Announce Type: replace Abstract: Growing demands from tasks like code generation, deep reasoning, and long-document understanding have made long-context generation a crucial capabi...