arXiv AI
Hyperparameter Scaling Laws Across MoE Sparsity
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Aug 21
Let's Scale Step by Step: Compute-Efficient Hyperparameter Transfer for Large-Scale Mixture-of-Experts
arXiv:2608. 20061v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) architectures significantly expand model capacity without a proportional increase in computational cost.
arXiv AI
Aug 12
Compute-Optimal Is Not Cluster-Optimal: Systems-Aware Scaling for Sparse Mixture-of-Experts
arXiv:2608. 10605v1 Announce Type: cross Abstract: In large-scale pretraining, the algorithm, architecture, and systems decisions are conventionally made in disconnected stages.
arXiv AI
Aug 5
LLaDA MoE v2: Scaling Mixture-of-Experts Diffusion Language Models
arXiv:2608. 03457v1 Announce Type: new Abstract: Diffusion language models (dLLMs) offer an alternative to autoregressive (AR) language modeling, yet the scaling behavior of Mixture-of-Experts (MoE) dLLMs remains poorly understood.
arXiv AI
2d ago
Why Speculative Decoding Works Better Than Predicted on Sparse MoE Models
The paper investigates why speculative decoding (SD) performs better than expected on sparse mixture-of-experts (MoE) models. By evaluating 10 large MoE models, the authors find that recent fine‑grained models achieve higher speedups at low batch sizes, contrary to prior predictions of a concave speedup‑batch relationship. They attribute this to lower verification costs driven by temporal overlap of draft tokens and GPU execution dynamics, and develop a maximum‑entropy IPF framework that accurately predicts unique expert counts from routing overlap, closely matching real‑model behavior.
arXiv AI
Jun 2
When Data Is Scarce: Scaling Sparse Language Models with Repeated Training
arXiv:2606. 01155v1 Announce Type: cross Abstract: Scaling laws for dense LLMs under infinite data are well explored, but how sparsity interacts with limited data is not.
Hugging Face Trending Papers
Aug 11
Compute-Optimal Is Not Cluster-Optimal: Systems-Aware Scaling for Sparse Mixture-of-Experts
In large-scale pretraining, the algorithm, architecture, and systems decisions are conventionally made in disconnected stages. A scaling law stage selects an architecture and training recipe, optimizing loss under compute constraints, and a separate systems stage then optimizes the implementation for hardware efficiency.