The paper investigates speculative decoding for mixture‑of‑experts (MoE) language models, focusing on how deduplication of expert reads can reduce byte traffic and improve decoding speed. By modeling step time as a function of bytes requested and kernel bandwidth, the authors derive conditions under which speculation outperforms plain decoding and identify the optimal draft length as a staircase function of acceptance rate. Experiments on llama.cpp Metal, MLX‑LM, and llama.cpp CUDA show that deduplication is only faster for very short draft lengths (5–8 tokens) or not at all, and the predicted crossover points closely match measured savings across multiple configurations.
arXiv:2610.15417v1 Announce Type: new Abstract: Mixture-of-experts (MoE) models activate only a few experts per token, but when their expert weights are offloaded to host memory, host-to-device (H2D)...
The paper investigates how speculative decoding—where a draft model proposes multiple future tokens that are then verified by the target model—affects energy consumption in large language model serving under latency constraints. It finds that energy savings depend heavily on the acceptance rate of drafted tokens, which varies by model architecture, drafting scheme, and the type of text (e.g., conversation vs. code, English vs. Romansh). The study shows that while speculation can reduce energy per token at small batch sizes, at larger batches the verification cost dominates, potentially turning speculation into an energy loss, especially when acceptance rates are low.
The paper investigates why speculative decoding (SD) performs better than expected on sparse mixture-of-experts (MoE) models. By evaluating 10 large MoE models, the authors find that recent fine‑grained models achieve higher speedups at low batch sizes, contrary to prior predictions of a concave speedup‑batch relationship. They attribute this to lower verification costs driven by temporal overlap of draft tokens and GPU execution dynamics, and develop a maximum‑entropy IPF framework that accurately predicts unique expert counts from routing overlap, closely matching real‑model behavior.
arXiv:2607. 17283v1 Announce Type: new Abstract: Single-stream autoregressive decoding of large language models is bound by memory bandwidth: each generated token requires one full forward pass through the target model, and successive passes cannot be parallelized.
The paper introduces DuoRate, a method for allocating precision in Mixture-of-Experts (MoE) models that separates storage and decoding rates. It stores a uniform tier of weights once and then adapts read precision per workload using a Lagrangian rate‑distortion optimization, reducing the store‑read gap. Experiments on Qwen3-30B-A3B and GSM8K show that DuoRate achieves higher accuracy at lower bit‑per‑weight (bpw) rates compared to fixed‑format baselines.
arXiv:2607. 24434v1 Announce Type: cross Abstract: Large Mixture-of-Experts (MoE) language models are attractive for end-device deployment because only a small subset of experts is active per token, but their routed expert weights often exceed accelerator memory.
arXiv:2608.20530v1 Announce Type: new Abstract: Speculative decoding accelerates language-model inference by drafting future tokens that the target model verifies in parallel. A diffusion-style block...
arXiv:2609.36222v1 Announce Type: new Abstract: Large language models are increasingly expensive to serve. In large-scale serving systems, autoregressive decoding is often bottlenecked by transferrin...
Large Mixture-of-Experts (MoE) language models are attractive for end-device deployment because only a small subset of experts is active per token, but their routed expert weights often exceed accelerator memory. We target latency-critical single-user settings where routed experts are staged on demand from CPU memory to a GPU or from Flash to a mobile NPU.
arXiv:2607. 21535v1 Announce Type: new Abstract: Speculative decoding accelerates autoregressive generation by having a cheap draft propose tokens that a target verifies in parallel.
arXiv:2609.37532v1 Announce Type: cross Abstract: Growing large language model applications demand efficient inference. At high concurrency, block-diffusion speculative decoding suffers from verifica...
The paper investigates whether training Mixture-of-Experts (MoE) routers can improve memory‑bandwidth locality on consumer GPUs. Using a new zero‑surgery telemetry tool, the authors measure that a large Qwen3‑235B model is bottlenecked by disk‑based expert access, and that an LRU cache can serve a majority of requests. They pre‑register experiments training 137 M‑parameter MoE models with locality‑aware losses, finding that while cache misses can drop up to 60 % (99 % static‑pin hit rate), every configuration fails to meet a strict 1 % perplexity threshold, indicating a tight coupling between cache efficiency and model quality.