arXiv Machine Learning
Structure Aware Neural Architecture Search for Mixture of Experts
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv AI
Aug 21
Towards Efficient Pareto Set Approximation via Mixture of Experts Based Model Fusion
arXiv:2406. 09770v2 Announce Type: replace-cross Abstract: Solving multi-objective optimization problems for large deep neural networks is a challenging task due to the complexity of the loss landscape and the expensive computational cost of training and evaluating models.
arXiv AI
Sep 10
MoEMB: Scaling Universal Multimodal Embeddings with Efficient Mixture-of-Experts Models
MoEMB introduces a mixture‑of‑experts (MoE) approach to scale universal multimodal embeddings (UME) without increasing the size of the output vector or relying on autoregressive decoding. By expanding encoder capacity along the expert axis, MoEMB achieves state‑of‑the‑art performance on MMEB‑V2 and MRMR benchmarks with only 3 B active parameters, outperforming TTE‑based methods that use more than four times as many active parameters and require significantly more compute. The paper also presents the first comprehensive study of adaptive computation for MoE‑based embeddings, exploring training‑time and inference‑time strategies to further improve efficiency for large‑scale retrieval and recommendation systems.
arXiv AI
Oct 15
SLEM: Spectrum-Aware Latent Expert Merging
arXiv:2610.16484v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) models scale model capacity efficiently by activating only a small subset of experts for each input, while still requiring a...
arXiv AI
Oct 21
Reconstruction-Optimized Expert Pruning for Mixture-of-Experts Language Models
arXiv:2610.27123v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) models reduce computation through sparse expert activation, but deployment still requires storing a large number of expert p...
Hugging Face Trending Papers
Jul 5
Optimal Mixture-of-Experts Model Averaging for Conditional Generative Models
Conditional generative models have emerged as powerful tools for sampling from target conditional distributions, driving substantial advances across a wide range of scientific and applied domains. As these models proliferate, practitioners often face multiple plausible generators whose performance can vary with the task, data, or input condition.
arXiv Machine Learning
Jun 25
Learning Subset-Shared Invariances for Domain Generalization with Mixture-of-Experts
arXiv:2606. 25665v1 Announce Type: new Abstract: Domain generalization (DG) aims to learn a model from one or more source domains that generalizes to an unseen target domain without accessing target data during training.