← Back to all news
arXiv Computation and Language By Avni Mittal, Avinash Anand, Ashutosh Kumar, Dikshant Kukreja, Kritarth Prasad, Chaitanya Lakhchaura, Rishitej Reddy Vyalla, Erik Cambria, Zhengkui Wang, Rajiv Ratn Shah

Sonar-MoE: In-Backward Expert Updates for Memory-Efficient Mixture-of-Experts Training

Read the original on arXiv Computation and Language →

Sonar‑MoE introduces a memory‑efficient training technique for Mixture‑of‑Experts models by updating each expert tensor during the backward pass and immediately freeing its gradient, ensuring that GPU memory never holds more than one layer of expert gradients. In experiments fine‑tuning OLMoE‑1B‑7B and Granite‑3.1‑3B‑A800M in bf16, Sonar‑MoE reduces peak memory usage by 19% to 22% and speeds up each training step by 17% to 23% compared to standard training with global‑norm clipping. The method achieves these gains while maintaining bitwise identical updates to standard training (without expert gradient clipping) and keeping held‑out loss within 0.003 of all baselines.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv Machine Learning
Oct 12

Monarch-MoE: Structured Experts for Multimodal Continual Instruction Tuning

Monarch-MoE introduces structured experts for multimodal continual instruction tuning by replacing dense LoRA factors with block‑diagonal ones linked by a fixed permutation, enabling blockwise gradient isolation. This design reduces adapter parameters by up to 75% and 50% on two benchmarks while keeping first‑task forgetting below 0.52 and 0.10 percentage points, respectively. The method is optimized for memory‑limited hardware, allowing fine‑tuning and inference on a single 24 GB RTX 4090 with lower memory usage and competitive runtime.

By Xi Zhang, Boyu Diao, Zesen Chen, Ruiqi Liu, Hangda Liu, Yongjun Xu
fine-tuningmultimodal
More like this →
arXiv AI
Oct 5

MaskCoFT: Masked Co-Adaptive Fine-Tuning for Memory-Efficient MoE Inference

MaskCoFT introduces a masked co‑adaptive fine‑tuning approach for mixture‑of‑experts language models, training both routers and experts jointly with a cross‑entropy loss. A learnable binary mask limits each layer’s Top‑K routing to a subset of experts during fine‑tuning, allowing experts to adapt to the tokens they receive. In simulated GPU cache scenarios, MaskCoFT reduces expert fetches per token by 23.7% for Mixtral‑8x7B and 10.1% for DeepSeek‑V2‑Lite, and lowers inference time per output token by up to 16.4% and 5.5% respectively, while maintaining or improving accuracy across nine benchmarks.

By Junfeng Wu, Zehao Fan, Hadjer Benmeziane, Kaoutar El Maghraoui, Liu Liu, Yinan Wang
llmsfine-tuningbenchmarks
More like this →