arXiv AI
SpecFit: Not All Draft Tokens Cost the Same in Offloaded MoE Inference
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Jul 28
DraftExpert: Expansion-Aware Self-Speculative Decoding for End-Device MoE Inference
arXiv:2607. 24434v1 Announce Type: cross Abstract: Large Mixture-of-Experts (MoE) language models are attractive for end-device deployment because only a small subset of experts is active per token, but their routed expert weights often exceed accelerator memory.
Hugging Face Trending Papers
Jul 27
DraftExpert: Expansion-Aware Self-Speculative Decoding for End-Device MoE Inference
Large Mixture-of-Experts (MoE) language models are attractive for end-device deployment because only a small subset of experts is active per token, but their routed expert weights often exceed accelerator memory. We target latency-critical single-user settings where routed experts are staged on demand from CPU memory to a GPU or from Flash to a mobile NPU.
arXiv AI
Jul 15
Less Experts, Faster Decoding: Cost-Aware Speculative Decoding for Mixture-of-Experts
arXiv:2607. 12696v1 Announce Type: cross Abstract: Sparse Mixture-of-Experts (MoE) models have become an important approach for scaling Large Language Models (LLMs), but their inference efficiency depends strongly on expert activation patterns.
arXiv Machine Learning
Aug 5
AcceptMoE: Commitment-Weighted Self-Sizing Verifier Expert Sets for Efficient MoE Speculative Decoding
arXiv:2608. 02989v1 Announce Type: new Abstract: Speculative decoding verifies a tree of draft tokens in one target-model forward pass.
arXiv Machine Learning
Sep 1
Ceiling-Clipped Acceptance Histograms Indicate Stranded Speed-up in Block-Diffusion Speculative Decoding
arXiv:2608.30427v1 Announce Type: cross Abstract: Speculative decoding speeds up generation with an efficient draft model (drafter) that proposes tokens for a target model to verify in one pass, pres...
arXiv AI
Aug 5
Oilbird: Training-Free Speculative Decoding with Keys the Verifier Already Computes
arXiv:2608. 03839v1 Announce Type: new Abstract: Training-free speculative decoding drafts by matching an exact suffix of the context against a pool of earlier context.