The paper presents a statistical framework for Mixture-of-Experts (MoE) models, treating them as localized aggregation systems. It derives oracle risk bounds that separate approximation, expert‑learning, and router‑estimation errors for both dense and sparse routing with evolving experts. The authors also analyze how sparse Top‑K routing balances computational cost with performance, interpret gating geometrically, and explain how shared experts can capture common predictive structure while allowing routed experts to focus on local residuals.
By Siyuan He, Bokai Yang, Jie Hu, Ziwen Gao, Yuhong Yang
arXiv:2606. 17419v1 Announce Type: new Abstract: We develop approximation and generalization error estimates for multi-input neural operators, with the output error measured in Sobolev norms.
By Yahong Yang, Zecheng Zhang, Wei Zhu, Wenjing Liao, Hao Liu
arXiv:2608. 15982v1 Announce Type: new Abstract: We develop operator-theoretic generalization bounds for deep multi-output function classes by representing network layers as Koopman composition operators on vector-valued reproducing kernel Hilbert spaces.
By Mahdi Mohammadigohari, Thomas Borsani, Giuseppe Di Fatta
arXiv:2609. 05263v1 Announce Type: cross Abstract: We investigate the best $L_2$ approximation of mixed Sobolev spaces by shallow neural networks with $n$ neurons and general activation functions.
By Yuwen Li, Guozhi Zhang
arXiv:2607. 06781v1 Announce Type: new Abstract: In this work, we investigate the fixed-architecture neural network approximation with explicit parameter bounds and elementary activations.
By Feng-Lei Fan, Ze-Yu Li, Chen-Yu Wang, Jian-Jun Wang
We develop operator-theoretic generalization bounds for deep multi-output function classes by representing network layers as Koopman composition operators on vector-valued reproducing kernel Hilbert spaces. In vector-valued Sobolev RKHSs, we derive Rademacher complexity bounds for invertible and width-expanding injective architectures.
arXiv:2503. 24092v2 Announce Type: replace-cross Abstract: Motivated by the rapidly growing field of mathematics for operator approximation with neural networks, we present a novel universal operator approximation theorem for broad classes of encoder-decoder architectures and a wide range of input and output spaces.
By Janek G\"odeke, Pascal Fernsel
arXiv:2606. 26705v1 Announce Type: cross Abstract: Feedforward neural network (NN) expressivity is typically studied by emulating optimal basis-expansion schemes.
By Anastasis Kratsios, Simone Brugiapaglia, Bum Jun Kim, Gregory Cousins, Haitz S\'aez de Oc\'ariz Borde
arXiv:2602. 06154v2 Announce Type: replace Abstract: Mixture-of-Experts (MoE) models scale large language models efficiently by sparsely activating experts, but once an expert is selected, it is executed fully.
By Nurbek Tastan, Stefanos Laskaridis, Karthik Nandakumar, Samuel Horvath
Recent studies have shown that smooth functions can be well approximated by ReLU neural networks with path norm constraint on the weights. We extend these results from uniform approximation to approximation in Sobolev norm.
The paper introduces L2R, a routing framework for Mixture-of-Experts models that reshapes the routing space into a shared low‑rank latent space and employs Saturated Inner‑Product Scoring to control Lipschitz behavior, resulting in smoother and more stable routing geometry. It also adds a parameter‑efficient multi‑anchor routing mechanism to increase expert expressiveness. Experiments on an OLMoE‑based language model and a ViT‑based ImageNet setting demonstrate improved overall performance and better routing geometry and expert discrimination.
By Minghao Yang, Ren Togo, Guang Li, Takahiro Ogawa, Miki Haseyama
In this work, we investigate the fixed-architecture neural network approximation with explicit parameter bounds and elementary activations. While prior work demonstrated super-expressive approximation using fixed-size networks, they lack quantitative and non-asymptotic characterizations of parameter magnitude with respect to the approximation error.