FastAFD is an attention‑FFN disaggregation engine designed for rack‑scale NVLink systems such as GB200 NVL72. It overlaps the mixture‑of‑experts computation of one microbatch with the attention of another, achieving up to 1.17× higher throughput than the best wide‑expert‑parallelism vLLM configuration on Qwen3 and Qwen3.5, and matching it on MiniMax‑M2.5. An analytical model predicts AFD performance within 3% using only colocated vLLM profiles, revealing that speedup comes from batch and overlap gains that outweigh disaggregation costs, and that speculative decoding further boosts performance for light‑expert models.
arXiv:2607. 21535v1 Announce Type: new Abstract: Speculative decoding accelerates autoregressive generation by having a cheap draft propose tokens that a target verifies in parallel.
The paper investigates the performance impact of using a remote KV cache tier for long‑context language model serving. In a disaggregated deployment of GLM‑5.3 on 8xB300 nodes, enabling the remote tier initially increased the 16k‑token p99 latency from 3.96 s to 17.3 s, and synchronous loading caused severe queue instability. By sharding the KV store across context‑parallel ranks, partitioning the remote tier, adding asynchronous loading, and fixing client bugs, the authors reduced the p99 latency from 83.32 s to 0.67 s and increased prefill throughput by 14–34 %. whyItMatters":"The study demonstrates that careful sharding and asynchronous handling of remote KV caches can dramatically improve latency and throughput in large‑scale LLM serving systems."
Prefetch‑Steered Routing (PSR) is a deterministic, cache‑aware method for selecting experts in Mixture‑of‑Experts models that limits physical weight copies to a logical cache of at most *n* experts per layer and token. On several large‑scale models (OLMoE‑1B‑7B, Moonlight‑16B‑A3B, Qwen3.6‑35B‑A3B) PSR achieves 2–2.3× faster decoding than exact routing with LRU and outperforms the Cache‑Prior baseline by 25–30% in many benchmarks, while maintaining comparable accuracy. The approach also shows that speculative block‑level expert selection can be faster than autoregressive decoding when GPU memory is constrained.
Speculative decoding accelerates autoregressive generation by having a cheap draft propose tokens that a target verifies in parallel. Frontier models increasingly ship a built-in Multi-Token-Prediction (MTP/NEXTN) draft head under the assumption that the draft is negligibly cheap.
arXiv:2605. 15422v3 Announce Type: replace Abstract: Modern RL post-training methods such as GRPO and DAPO train on N response sequences of R tokens sampled from a shared prompt of P tokens, but standard FlashAttention replicates all P prompt tokens N times across both forward and backward passes -- duplicating compute and memory on identical hidden states.
arXiv:2609.37626v1 Announce Type: cross Abstract: No single way of parallelizing attention serves large language models well under all loads. Low concurrency favors tensor parallelism, many independe...
The paper introduces Group-Query Latent Attention (GQLA), a variant of Multi-head Latent Attention that shares group‑sharded up‑projections to enable a joint latent representation and a shardable expanded cache. GQLA supports both MQA‑absorb and GQA execution using the same weights and existing kernels, and roofline analysis shows how absorption trades traffic for computation while GQA shards KV groups. Experiments on DeepSeek‑V3.1‑Base and other models demonstrate that GQLA achieves comparable training losses to MLA, maintains high accuracy after checkpoint conversion, and improves output throughput by up to 35% on eight H20‑3e GPUs, while also allowing larger batch sizes due to a smaller per‑rank cache.
arXiv:2607. 05876v1 Announce Type: cross Abstract: LLM serving optimization typically benchmarks many configurations and reaches for heavy profilers when latency targets are missed.
WiSP (Working‑Set Paging) is a routing‑aware expert pager that allows Mixture‑of‑Experts models to run on GPUs that cannot hold the entire expert pool by paging experts in and out of VRAM while preserving byte‑identical outputs. On a 24 GiB RTX 3090, WiSP doubles decode throughput compared to static offload when the model does not fit, and its companion policy MV‑WSA allocates VRAM between resident experts and KV cache based on marginal latency benefit, reducing end‑to‑end time by up to 1.19× without altering model outputs.
arXiv:2608. 13057v1 Announce Type: cross Abstract: In expert-parallel (EP) MoE serving, every layer synchronizes at the slowest GPU.
arXiv:2608. 04502v1 Announce Type: cross Abstract: Attention--Feed-Forward Network (FFN) Disaggregation (AFD) is emerging as a promising architecture for serving Mixture-of-Experts (MoE) language models.
The paper investigates whether training Mixture-of-Experts (MoE) routers can improve memory‑bandwidth locality on consumer GPUs. Using a new zero‑surgery telemetry tool, the authors measure that a large Qwen3‑235B model is bottlenecked by disk‑based expert access, and that an LRU cache can serve a majority of requests. They pre‑register experiments training 137 M‑parameter MoE models with locality‑aware losses, finding that while cache misses can drop up to 60 % (99 % static‑pin hit rate), every configuration fails to meet a strict 1 % perplexity threshold, indicating a tight coupling between cache efficiency and model quality.