arXiv AI
Scaling Laws for Looped Mixture of Experts
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Jun 4
LoopMoE: Unifying Iterative Computation with Mixture-of-Experts for Language Modeling
arXiv:2606. 04438v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) and looped architectures scale models along two orthogonal axes, namely parameter capacity and effective depth.
arXiv Machine Learning
Sep 2
SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers
arXiv:2609.01343v1 Announce Type: new Abstract: Looped Transformers increase effective depth by iterating a shared block of layers, but most evaluations compare at fixed model size, conflating archit...
arXiv Machine Learning
Jul 1
Sparse Layers are Critical to Scaling Looped Language Models
arXiv:2605. 09165v2 Announce Type: replace Abstract: Looped language models repeat a set of transformer layers through depth, reducing memory costs and providing natural early-exit points at loop boundaries.
arXiv AI
Sep 10
Hyperparameter Scaling Laws Across MoE Sparsity
arXiv:2609.08690v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) models expand model capacity without a proportional increase in training compute, but increasing sparsity makes reliable hyp...
arXiv AI
Jul 28
cMoLLM at Scale: Horizontal Scaling Laws for Mixture-of-LLMs
arXiv:2607. 22577v1 Announce Type: new Abstract: Scaling large language models (LLMs) has driven their success, yet dense Transformers couple capacity and computation: every parameter is activated for every token, making training and inference costs grow linearly with model size-a critical bottleneck as models approach trillion-parameter regimes.
arXiv Machine Learning
2d ago
What Makes Recurrence Effective in Looped Language Models?
arXiv:2609.36636v1 Announce Type: new Abstract: Looped language models (LoopLMs) increase computational depth through parameter sharing, offering a path to scale inference computation without adding...