HAPMoE: Heterogeneity-Aware Automatic Parallelism Planning for Mixture-of-Experts Models Training
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
The Flow has not summarised this story yet — read it at arXiv Machine Learning.
As model sizes continue to scale, distributed training has become inevitable. Automatic parallelization techniques can derive efficient training parallelism strategies at low cost while achieving supe...
arXiv:2509. 23722v2 Announce Type: replace-cross Abstract: Pipeline parallelism is widely used to train large language models (LLMs).
arXiv:2607. 01844v1 Announce Type: cross Abstract: This paper showcases a memory-efficient training stack for Mixture-of-Experts (MoE) models.
arXiv:2601. 20408v2 Announce Type: replace-cross Abstract: Enterprise LLM deployment faces a critical scalability challenge: organizations must optimize models systematically to scale AI initiatives within constrained compute budgets, yet the specialized expertise required for manual optimization remains a niche and scarce skillset.
LM‑SimBench is a unified framework, dataset, and benchmark designed to evaluate large language model training simulators. It decomposes simulation into functional stages—from model compilation to schedule‑level aggregation—and connects execution representations with cost estimators to measurable targets. The authors profiled 23 model settings across 10 dense and Mixture‑of‑Experts families on an Ascend NPU cluster, producing a dataset of 348 feasible configurations that vary five parallelism dimensions and activation recomputation, and used it to compare simulators, assess configuration selection, and diagnose prediction errors, with cross‑platform validation on GPUs.
arXiv:2606. 19025v1 Announce Type: cross Abstract: Pre-training Large Language Models (LLMs) typically demands large-scale infrastructure with tightly coupled hardware accelerators.