arXiv Machine Learning
LatentLoop: Scaling Looped MoE Models with Asymmetric Compute Allocation
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv AI
Oct 1
Scaling Laws for Looped Mixture of Experts
arXiv:2609.40316v1 Announce Type: cross Abstract: Looped transformers and Mixture-of-Experts (MoE) offer complementary routes to efficient scaling: recurrence increases computational depth at fixed p...
arXiv AI
Jun 4
LoopMoE: Unifying Iterative Computation with Mixture-of-Experts for Language Modeling
arXiv:2606. 04438v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) and looped architectures scale models along two orthogonal axes, namely parameter capacity and effective depth.
arXiv AI
Oct 2
Looping Beyond Twice: A Scalable Recipe for Looped Mixture-of-Experts
The paper introduces LOOM, a method for scaling looped mixture‑of‑experts (MoE) Transformers beyond the typical two‑loop limit. LOOM addresses two key obstacles: it stabilizes deep recurrence by bounding residual variance and re‑injecting the input embedding, and it prevents expert selection collapse by using per‑loop routers and a looping residual to maintain computational diversity. Experiments on 100 M–1.7 B parameter models show stable scaling to 9–12 loops, with significant perplexity reductions and zero‑shot accuracy gains under near‑iso‑FLOP conditions.
arXiv Computation and Language
5d ago
One Model to Loop Them All: Elastic Latent Recurrence for Language Models
The paper introduces Elastic Latent Recurrence (ELR), a training recipe that enables looped language models to match the accuracy of non-recurrent models at a single loop while allowing additional loops to improve accuracy with more compute. ELR achieves this through gated state updates with learned step sizes, mixed loop counts during training, and conditioning on loop index and budget. The method also reduces training and inference overhead in Mixture-of-Experts models and produces a single checkpoint, Koi‑4B, that performs competitively with state‑of‑the‑art models on multiple benchmarks.
arXiv AI
2d ago
Virtual Head Attention
Virtual Head Attention (VHA) is a lightweight attention mechanism that improves long‑context large language models by adding two linear operations—Q Premix and Linear Postmix—to recover head diversity and fuse inter‑head features without nonlinearities. VHA reduces KV cache size, training and prefill FLOPs, and maintains or surpasses model quality compared to existing methods such as GQA, MLA, and MFA across multiple scales and benchmarks. The design allows VHA to be folded into standard GQA inference pipelines, enabling efficient deployment with only two KV heads and full exploitation of Tensor Core parallelism.
arXiv AI
Aug 12
Compute-Optimal Is Not Cluster-Optimal: Systems-Aware Scaling for Sparse Mixture-of-Experts
arXiv:2608. 10605v1 Announce Type: cross Abstract: In large-scale pretraining, the algorithm, architecture, and systems decisions are conventionally made in disconnected stages.