Sonar‑MoE introduces a memory‑efficient training technique for Mixture‑of‑Experts models by updating each expert tensor during the backward pass and immediately freeing its gradient, ensuring that GPU memory never holds more than one layer of expert gradients. In experiments fine‑tuning OLMoE‑1B‑7B and Granite‑3.1‑3B‑A800M in bf16, Sonar‑MoE reduces peak memory usage by 19% to 22% and speeds up each training step by 17% to 23% compared to standard training with global‑norm clipping. The method achieves these gains while maintaining bitwise identical updates to standard training (without expert gradient clipping) and keeping held‑out loss within 0.003 of all baselines.
arXiv:2608. 04407v1 Announce Type: cross Abstract: Memory-efficient matrix optimizers such as Sinkhorn gradient descent remove most AdamW optimizer state for dense Transformer matrices, but direct application to Mixture-of-Experts (MoE) training is unreliable.
arXiv:2609.36222v1 Announce Type: new Abstract: Large language models are increasingly expensive to serve. In large-scale serving systems, autoregressive decoding is often bottlenecked by transferrin...
arXiv:2502. 11034v3 Announce Type: replace Abstract: Loss spikes remain a persistent obstacle in large-scale language model pretraining.
arXiv:2607. 01844v1 Announce Type: cross Abstract: This paper showcases a memory-efficient training stack for Mixture-of-Experts (MoE) models.
Monarch-MoE introduces structured experts for multimodal continual instruction tuning by replacing dense LoRA factors with block‑diagonal ones linked by a fixed permutation, enabling blockwise gradient isolation. This design reduces adapter parameters by up to 75% and 50% on two benchmarks while keeping first‑task forgetting below 0.52 and 0.10 percentage points, respectively. The method is optimized for memory‑limited hardware, allowing fine‑tuning and inference on a single 24 GB RTX 4090 with lower memory usage and competitive runtime.
MaskCoFT introduces a masked co‑adaptive fine‑tuning approach for mixture‑of‑experts language models, training both routers and experts jointly with a cross‑entropy loss. A learnable binary mask limits each layer’s Top‑K routing to a subset of experts during fine‑tuning, allowing experts to adapt to the tokens they receive. In simulated GPU cache scenarios, MaskCoFT reduces expert fetches per token by 23.7% for Mixtral‑8x7B and 10.1% for DeepSeek‑V2‑Lite, and lowers inference time per output token by up to 16.4% and 5.5% respectively, while maintaining or improving accuracy across nine benchmarks.
arXiv:2607. 19058v1 Announce Type: cross Abstract: Optimizer state is the largest single line item in the memory budget of mixture-of-experts (MoE) training: on a 6.
arXiv:2608. 15383v1 Announce Type: new Abstract: Sparse mixture-of-experts (MoE) language models reduce arithmetic by activating only a small subset of experts per token, yet deployment still requires storing and moving the full expert bank.
arXiv:2607. 09686v1 Announce Type: new Abstract: Sparse Mixture-of-Experts (MoE) language models separate total parameter count from per-token active computation, but local inference systems often still require the full model, key-value cache, runtime buffers, and operatingsystem headroom to fit in fast memory.
The paper introduces RelayMoE, a ring‑based execution model for Mixture‑of‑Experts (MoE) training that reduces memory usage by avoiding full top‑k expanded dispatch buffers. RelayMoE dynamically chooses between expert and token routing based on communication volume and overlaps data transfer with computation, enabling efficient backward recomputation. Experiments on 30B–57B MoE models show up to a 2× speedup over Megatron‑LM and a 2.02× throughput improvement while extending trainable sequence lengths by up to 2.85×.
arXiv:2607. 24434v1 Announce Type: cross Abstract: Large Mixture-of-Experts (MoE) language models are attractive for end-device deployment because only a small subset of experts is active per token, but their routed expert weights often exceed accelerator memory.
The paper introduces ALTO, an open‑source training recipe that enables a 20‑billion‑parameter mixture‑of‑experts LLM (GPT‑OSS‑20B) to train on the MLPerf Small MoE benchmark using the MXFP4 sub‑8‑bit format. By combining hybrid 1D/2D block quantization, randomized Hadamard transforms, stochastic rounding, and a simplified weight de‑oscillation scheme, the authors achieve a validation loss of 3.3350 after 16,128 steps—only 0.007 above the BF16 baseline—while reducing the FP4 convergence overhead from +20.0% to +5.0% relative to BF16. The study also reports negative results for differential gradient estimation, outlier clipping, and macro‑block scaling, despite operator‑level SNR improvements, and notes that the experiments use simulated MXFP4 kernels on AMD MI300 hardware without wall‑clock optimization.