arXiv:2607. 21535v1 Announce Type: new Abstract: Speculative decoding accelerates autoregressive generation by having a cheap draft propose tokens that a target verifies in parallel.
By Alagappan Valliappan
The paper investigates the performance impact of using a remote KV cache tier for long‑context language model serving. In a disaggregated deployment of GLM‑5.3 on 8xB300 nodes, enabling the remote tier initially increased the 16k‑token p99 latency from 3.96 s to 17.3 s, and synchronous loading caused severe queue instability. By sharding the KV store across context‑parallel ranks, partitioning the remote tier, adding asynchronous loading, and fixing client bugs, the authors reduced the p99 latency from 83.32 s to 0.67 s and increased prefill throughput by 14–34 %.
whyItMatters":"The study demonstrates that careful sharding and asynchronous handling of remote KV caches can dramatically improve latency and throughput in large‑scale LLM serving systems."
By Muhammad Ibrahim
Prefetch‑Steered Routing (PSR) is a deterministic, cache‑aware method for selecting experts in Mixture‑of‑Experts models that limits physical weight copies to a logical cache of at most *n* experts per layer and token. On several large‑scale models (OLMoE‑1B‑7B, Moonlight‑16B‑A3B, Qwen3.6‑35B‑A3B) PSR achieves 2–2.3× faster decoding than exact routing with LRU and outperforms the Cache‑Prior baseline by 25–30% in many benchmarks, while maintaining comparable accuracy. The approach also shows that speculative block‑level expert selection can be faster than autoregressive decoding when GPU memory is constrained.
By Siming Huang
Speculative decoding accelerates autoregressive generation by having a cheap draft propose tokens that a target verifies in parallel. Frontier models increasingly ship a built-in Multi-Token-Prediction (MTP/NEXTN) draft head under the assumption that the draft is negligibly cheap.
arXiv:2605. 15422v3 Announce Type: replace Abstract: Modern RL post-training methods such as GRPO and DAPO train on N response sequences of R tokens sampled from a shared prompt of P tokens, but standard FlashAttention replicates all P prompt tokens N times across both forward and backward passes -- duplicating compute and memory on identical hidden states.
By Jiading Gai, Shuai Zhang, Xiang Song, Bernie Wang, George Karypis
arXiv:2609.37626v1 Announce Type: cross
Abstract: No single way of parallelizing attention serves large language models well under all loads. Low concurrency favors tensor parallelism, many independe...
By Chuan Liu, Shuoming Zhang, Zhicheng Li, Qianqi Sun, Ruiyuan Xu, Qiuchu Yu, Xiyu Shi, Huimin Cui, Jiacheng Zhao