arXiv Machine Learning
Mixture of Many Zero-Compute Experts: A High-Rate Quantization Theory Perspective
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Jun 4
AlphaQ: Calibration-Free Bit Allocation for Mixture-of-Experts Quantization
arXiv:2606. 04980v1 Announce Type: new Abstract: Mixture-of-Experts (MoE) architectures scale model capacity through sparse expert activation, but their deployment remains memory-bound because all expert weights must reside in memory.
arXiv Machine Learning
Sep 4
Towards a Statistical Understanding of Mixture-of-Experts
The paper presents a statistical framework for Mixture-of-Experts (MoE) models, treating them as localized aggregation systems. It derives oracle risk bounds that separate approximation, expert‑learning, and router‑estimation errors for both dense and sparse routing with evolving experts. The authors also analyze how sparse Top‑K routing balances computational cost with performance, interpret gating geometrically, and explain how shared experts can capture common predictive structure while allowing routed experts to focus on local residuals.
arXiv Statistics ML
Sep 10
Regularized Estimation and Feature Selection in Mixtures of Generalized Linear Experts
arXiv:1907.06994v2 Announce Type: replace-cross Abstract: Mixtures of experts (MoE) are conditional mixture models in which both the mixing proportions and the component densities depend on the predi...
arXiv AI
5d ago
One-Shot Localisation of the Global Minimum of a Noisy One-Dimensional Function: An Iterative Neural Minimizer Compared with Set Transformers and Classical Estimators
The paper introduces the Neural Function Minimizer (NFM), an iterative neural model that predicts the global minimum of a noisy one‑dimensional function from only twenty samples. It compares the NFM to two Set Transformers and classical zero‑query estimators, finding that all learned models outperform classical methods on held‑out data, with the NFM achieving the lowest error relative to the Gaussian‑process posterior argmin. The study also examines how different estimator outputs affect tie‑breaking between equally deep valleys and plans to release a benchmark and self‑checking harness.
arXiv Machine Learning
Jun 17
MoSE: Mixture of Slimmable Experts for Efficient and Adaptive Language Models
arXiv:2602. 06154v2 Announce Type: replace Abstract: Mixture-of-Experts (MoE) models scale large language models efficiently by sparsely activating experts, but once an expert is selected, it is executed fully.
arXiv Machine Learning
Jun 9
Dendrograms of Mixing Measures for Softmax-Gated Gaussian Mixture of Experts: Consistency Without Model Sweeps
arXiv:2510. 12744v2 Announce Type: replace-cross Abstract: We develop a unified statistical framework for softmax-gated Gaussian mixture of experts (SGMoE) that addresses three long-standing obstacles in parameter estimation and model selection: (i) non-identifiability of gating parameters up to common translations, (ii) intrinsic gate-expert interactions that induce coupled differential relations in the likelihood, and (iii) the tight numerator-denominator coupling in the softmax-induced conditional density.