The paper introduces GRIP, an algorithm‑agnostic framework for machine unlearning in Mixture‑of‑Experts large language models. GRIP enforces hard geometric constraints on router updates, projecting gradient changes into the null space of the retain set’s routing matrix to prevent routing manipulation. Two variants—training‑time stochastic projection and post‑training analytical correction—show significant improvements in routing stability, retain accuracy, and resistance to white‑box adversarial recovery across two MoE models.
By Andy Zhu, Rongzhe Wei, Yupu Gu, Pan Li
arXiv:2604.03592v2 Announce Type: replace-cross
Abstract: Mixture-of-Experts (MoE) models exhibit striking performance disparities across languages, yet the internal mechanisms driving these gaps rem...
By Kening Zheng, Wei-Chieh Huang, Jiahao Huo, Zhonghao Li, Henry Peng Zou, Yibo Yan, Xin Zou, Jungang Li, Junzhuo Li, Hanrong Zhang, Xuming Hu, Philip S. Yu
The paper introduces RASET, a router‑agnostic safety‑critical expert tuning framework for Mixture‑of‑Experts (MoE) large language models. RASET identifies a small subset of experts that are responsible for safety enforcement and applies parameter‑efficient tuning only to those experts, preserving the model’s intrinsic routing behavior. Experiments on five open‑weight MoE backbones show that RASET achieves a high safety‑bypass yield, outperforming existing baselines by a significant margin.
By Zhibo Zhang, Yuxi Li, Zhen Ouyang, Ling Shi, Kailong Wang
arXiv:2610.07774v1 Announce Type: new
Abstract: Vision-language models (VLMs) face compositional safety risks where harmful intent emerges from the interaction between visual and textual inputs. As m...
By Ziyuan Yang, Wenxuan Ding, Shangbin Feng, Yulia Tsvetkov
arXiv:2604. 00421v2 Announce Type: replace Abstract: Mixture-of-Experts (MoE) layers increase model capacity by activating only a small subset of experts per token, and typically rely on a learned router to map hidden states to expert assignments.
By Jama Hussein Mohamud, Drew Wagner, Mirco Ravanelli
arXiv:2606.04160v2 Announce Type: replace-cross
Abstract: Safety alignment in instruction-tuned large language models (LLMs) depends on a model's ability to reliably refuse harmful or disallowed requ...
By Anna C. Marbut, Daniel R. Olson, Travis J. Wheeler
arXiv:2606. 10338v1 Announce Type: cross Abstract: Machine unlearning is increasingly important for large language models, yet unlearning in Mixture-of-Experts (MoE) architectures remains underexplored.
By Jingyi Xie, Yijun Lin, Yinjiang Xiong, Zhikun Zhang, Sai Li
The paper introduces Ban&Pick, a post‑training, plug‑and‑play routing strategy for Sparse Mixture‑of‑Experts large language models. It identifies and reinforces a small group of highly influential experts while dynamically pruning redundant ones, leading to accuracy gains across math, code, and reasoning benchmarks. Experiments on DeepSeek and Qwen3 show notable performance improvements and a 1.25× inference speedup without retraining or architectural changes.
By Yuanteng Chen, Peisong Wang, Yuantian Shao, Nanxin Zeng, Chang Xu, Jian Cheng
ACE introduces a training‑free, calibration‑free framework for adaptive expert skipping in Mixture‑of‑Experts LLMs. It combines a Global Spectral Proxy that estimates global transformation capacity with a Router‑Conditioned Refinement that builds expert‑specific direction prototypes, enabling the model to skip low‑contribution experts while always keeping the top‑1 expert. Offline computation of expert statistics leaves only lightweight table lookups during inference, and experiments on three MoE‑based LLMs show ACE outperforms static and dynamic baselines, especially at high skipping ratios.
By Zukang Xu, Zhixiong Zhao, Xing Hu, Jiangyong Yu, Houji Wen, Jun Li, Zhe Jiang, Dawei Yang
arXiv:2608. 06396v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) language models route each token through a small subset of experts, making routing patterns useful for identifying task-relevant experts during downstream adaptation.
By Guanzhi Deng, Haibo Wang, Kuan Wu, Xiangru Jian, Shing Yin Wong, Sichun Luo, Zhuoran Wang, Linqi Song
arXiv:2607. 08780v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) models activate only a sparse subset of experts per token, yet consecutive tokens frequently activate different experts -- causing constant weight swapping between slow storage and fast memory on edge devices.
By Ali Kayyam
The paper examines whether expert‑routing statistics from a Mixture‑of‑Experts language model can inform layer‑wise mixed‑precision quantization of routed‑expert feed‑forward networks. By building a routing‑capture pipeline for Qwen‑3.6‑35B‑A3B, the authors derive a composite sensitivity score and use coordinate descent to allocate bits per layer under a fixed budget. Experiments across flat, steep, and random allocation regimes show that routing‑guided reallocation does not outperform a uniform baseline, suggesting that routing behavior is an unreliable proxy for true quantization cost.
By Yifan Liu