Hugging Face Blog
Introducing Olmo-core 3: Open, scalable training infrastructure for large MoEs
Read the original on Hugging Face Blog →The Flow has not summarised this story yet — read it at Hugging Face Blog.
Related stories
arXiv AI
Sep 2
Instella-MoE Technical Report
Instella‑MoE is a fully open Mixture‑of‑Experts language model with 16 billion total parameters and 2.8 billion active parameters per token, trained from scratch on AMD Instinct GPUs. It incorporates a sparsely activated MoE design with Gated Multi‑head Latent Attention and FarSkip‑Collective connectivity, and follows a multi‑stage pipeline that includes pre‑training, long‑context extension, supervised fine‑tuning, direct preference optimization, and reinforcement learning with Multi‑Teacher On‑Policy Distillation. The model achieves an average score of 76.7 on pre‑training benchmarks and 73.2 on instruction‑following, reasoning, math, coding, and chat benchmarks, outperforming comparable fully open and open‑weight models, and its full training pipeline, weights, and code are released for reproducibility.
arXiv AI
Sep 17
The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction
The paper introduces Edge0, a streaming mixture‑of‑experts (MoE) inference engine that enables a 35‑billion‑parameter MoE model to run on consumer hardware by predicting routing decisions one token ahead. Edge0 uses a per‑layer prerouter to prefetch the necessary experts from SSD, and an unmerged recovery LoRA trained on the student path to recover quality lost to 4‑bit quantization and routing replacement. On a single 24‑GB machine, Edge0 serves the 35B MoE at 20 tokens per second while keeping peak active memory below 3 GiB, achieving performance close to its fp16 teacher across five public benchmarks.
arXiv AI
Jul 2
OmniMoE: An Efficient MoE by Orchestrating Atomic Experts at Scale
arXiv:2602. 05711v2 Announce Type: replace-cross Abstract: Mixture-of-Experts (MoE) architectures are evolving towards finer granularity to improve parameter efficiency.
arXiv Machine Learning
Aug 11
MoE-Prism: Disentangling Monolithic Experts for Elastic MoE Services via Model-System Co-Designs
arXiv:2510. 19366v2 Announce Type: replace-cross Abstract: Mixture-of-Experts (MoE) scales model capacity through sparse activation, and is becoming an important architecture for large language models (LLMs).
arXiv Machine Learning
2d ago
HAPMoE: Heterogeneity-Aware Automatic Parallelism Planning for Mixture-of-Experts Models Training
arXiv:2609.39350v1 Announce Type: cross Abstract: As model sizes continue to scale, distributed training has become inevitable. Automatic parallelization techniques can derive efficient training para...
arXiv AI
Sep 15
OpWeave: Flexible Operator Disaggregation for Heterogeneous LLM Serving
arXiv:2609.14237v1 Announce Type: cross Abstract: LLM serving systems increasingly disaggregate inference into finer-grained stages, with recent approaches separating attention from FFN or MoE execut...