arXiv Machine Learning
LoopSpec: Pipelined Self-Speculative Decoding for Looped Transformers
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
1d ago
WaveFront Decoding: Parallelized Self-Speculative Decoding for Looped Language Models
WaveFront Decoding (WFD) is a training‑free, self‑speculative decoding framework for looped language models that reduces decoding latency by batching draft and verification steps within the same recurrent‑block calls. By exploiting intermediate recurrence outputs as draft predictions and weight sharing to process token states at different depths together, WFD arranges mixed‑depth states into a diagonal wavefront, allowing shallow‑depth drafting while deeper‑depth verification proceeds concurrently. Experiments on six Spec‑Bench task categories show WFD achieving up to 4.81× speedup on Huginn‑3.5B compared to autoregressive decoding, outperforming traditional draft‑then‑verify approaches.
arXiv Machine Learning
Jul 13
Accelerating Large Language Model Inference with Self-Supervised Early Exits
arXiv:2407. 21082v3 Announce Type: replace-cross Abstract: This paper presents a modular approach to accelerate inference in large language models (LLMs) by adding early exit heads at intermediate transformer layers.
arXiv Machine Learning
Aug 27
Gated Recurrent Transformers: Expressive Depth through Recurrent Modulation
The paper introduces Gated Recurrent Transformers, a depth‑sharing architecture that brackets a single shared core with fixed prelude and coda blocks and uses a lightweight projection and element‑wise update gate to modulate recurrent updates. This design allows functional specialization across recurrences while reducing memory footprint. Experiments show that, under equal FLOPs or parameter budgets, the recurrent model matches or surpasses deeper GPT‑2 Small baselines, achieving similar or better accuracy with fewer parameters and lower peak decoding memory.
arXiv AI
Jul 10
A Practical Investigation of Training-free Relaxed Speculative Decoding
arXiv:2607. 08690v1 Announce Type: cross Abstract: Speculative decoding accelerates sampling from an autoregressive LLM by using a faster auxiliary model to draft tokens which are then verified in parallel by the LLM.
Hugging Face Trending Papers
Jul 9
A Practical Investigation of Training-free Relaxed Speculative Decoding
Speculative decoding accelerates sampling from an autoregressive LLM by using a faster auxiliary model to draft tokens which are then verified in parallel by the LLM. Standard speculative decoding is lossless: its rejection and resampling steps exactly preserve the LLM's sampling distribution.
arXiv Machine Learning
Aug 26
Gated Recurrent Transformers: Expressive Depth through Recurrent Modulation in Transformers
arXiv:2608.15062v3 Announce Type: replace-cross Abstract: Scaling transformer language models creates an inherent tension between expressivity and memory efficiency. While unique weights across layer...