arXiv Machine Learning

HAPMoE: Heterogeneity-Aware Automatic Parallelism Planning for Mixture-of-Experts Models Training

arXiv AI
Jun 9

Meeting SLOs, Slashing Hours: Automated Enterprise LLM Optimization with OptiKIT

arXiv:2601. 20408v2 Announce Type: replace-cross Abstract: Enterprise LLM deployment faces a critical scalability challenge: organizations must optimize models systematically to scale AI initiatives within constrained compute budgets, yet the specialized expertise required for manual optimization remains a niche and scarce skillset.

By Nicholas Santavas, Kareem Eissa, Patrycja Cieplicka, Piotr Florek, Matteo Nulli, Stefan Vasilev, Seyyed Hadi Hashemi, Antonios Gasteratos, Shahram Khadivi
arXiv AI
6d ago

LM-SimBench: A Unified Framework, Dataset, and Benchmark for Large Language Model Training Simulation

LM‑SimBench is a unified framework, dataset, and benchmark designed to evaluate large language model training simulators. It decomposes simulation into functional stages—from model compilation to schedule‑level aggregation—and connects execution representations with cost estimators to measurable targets. The authors profiled 23 model settings across 10 dense and Mixture‑of‑Experts families on an Ascend NPU cluster, producing a dataset of 348 feasible configurations that vary five parallelism dimensions and activation recomputation, and used it to compare simulators, assess configuration selection, and diagnose prediction errors, with cross‑platform validation on GPUs.

By Yu Dong, Hong Li, Qiang Sun, Xu Zhou, Jinbiao Huo, Qixiu Cheng, Zhiyuan Liu
arXiv Machine Learning
Oct 16

ETH: Data-Balanced Heterogeneous Pipeline Parallelism for Training Large Models

The paper introduces ETH, a training system that implements Heterogeneous Pipeline Parallelism (HPP) to achieve fine‑grained data balance across heterogeneous pipelines. ETH uses a hierarchical workload balance framework and block‑wise gradient aggregation to reduce communication overhead, while also providing optimizations that cut configuration search time. Experiments show ETH can double training speed and cut configuration generation overhead by up to 56× compared to existing systems.

By Mianjie Yu, Zhongmin Zhao, Zizhao Mo, Jianxiong Liao, Wenxuan Li, Yulin Qiao, Huanle Xu, Wenquan Yang, Yongqiang Yang, Zeren Li, Ruifeng Tang, Chengzhong Xu
arXiv AI
Jun 10

Piper: A Programmable Distributed Training System

arXiv:2606. 11169v1 Announce Type: cross Abstract: Large-scale model training increasingly relies on composing multiple parallelism strategies, such as data, pipeline, and expert parallelism, together with memory-saving optimizations like ZeRO.

By Megan Frisella, Shubham Tiwari, Andy Ruan, Yi Pan, Parker Gustafson, Mat Jacob, Gilbert Bernstein, Stephanie Wang
arXiv AI
3d ago

Beyond the Trial-and-Error Loop: Hybrid Projection and Automated Tuning for Distributed Training

The paper introduces Primus Projection, a hybrid engine that combines GPU microbenchmarks with analytical scaling to predict multi‑node distributed training performance, and Primus Tuning Agent, an automated search that uses the projection engine as a fast oracle to explore the vast configuration space of parallelism and micro‑batching. Together, they reduce the need for costly trial‑and‑error on hundreds of GPUs, achieving predictions within ~10% of real throughput and enabling significant throughput gains—up to +27% in a Mixtral 8x22B case study—within minutes of single‑node exploration.

By Anshu Raina, Peyman Razaghi, Yuankai Chen, Cheng Yao, Yao Fu, Vidushi Goyal, Devang Patel, Wen Xie, Zhenyu Gu, Emad Barsoum