The study examines whether Mixture-of-Experts (MoE) language models develop linguistically structured expert routing during bilingual language acquisition. Using a decoder-only English‑German MoE Transformer trained with sequential language exposure, the authors probe token‑level routing distributions and measure category‑dependent specialization via mutual information, routing entropy, and Jensen‑Shannon distance. Results show that a curriculum‑trained model peaks at a mutual information of 0.1148 at layer 5, while a no‑curriculum baseline trained on mixed data achieves a higher peak of 0.2599 at the same layer, indicating stronger aggregate specialization. Replication with a second training seed reveals that the no‑curriculum condition’s specialization focuses on a single language in a seed‑dependent way, whereas the curriculum consistently yields a stable, language‑balanced routing profile, suggesting that staged bilingual exposure reduces single‑language dominance.
By Amrit Gopinath, Raghul, Durairaj Thenmozhi
We investigate whether Mixture-of-Experts (MoE) language models develop linguistically structured expert routing during bilingual language acquisition. Inspired by the Declarative-Procedural framework...
The paper introduces RARE, a router‑agnostic representation engineering framework for Mixture‑of‑Experts language models. RARE projects behavioral perturbations onto the null space of the router matrix to avoid affecting routing, and corrects downstream routing drift. Experiments on six open‑weight MoE models show that RARE improves steering tasks—reducing harmfulness, increasing truthfulness, and enhancing factual editing—while preserving overall model accuracy.
By Zhibo Zhang, Zhen Ouyang, Ling Shi, Kailong Wang
arXiv:2608. 06396v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) language models route each token through a small subset of experts, making routing patterns useful for identifying task-relevant experts during downstream adaptation.
By Guanzhi Deng, Haibo Wang, Kuan Wu, Xiangru Jian, Shing Yin Wong, Sichun Luo, Zhuoran Wang, Linqi Song
arXiv:2411. 00918v5 Announce Type: replace-cross Abstract: Mixture of experts (MoE) architectures have become a cornerstone for scaling up and are a key component in most large language models such as GPT-OSS, DeepSeek-V3, Llama-4, and Gemini-2.
By Nam V. Nguyen, Thong T. Doan, Luong Tran, Van Nguyen, Quang Pham
arXiv:2606. 16456v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) models enable efficient scaling, but training them from scratch remains prohibitively expensive.
By Weiqiao Shan, Ruixiang Mao, Yuang Li, Yuhao Zhang, Yingfeng Luo, Tong Zheng, Chen Xu, Yucheng Qiao, Chunxiang Jin, Yi Yuan, Jingdong Chen, Tong Xiao, Jingbo Zhu
arXiv:2606. 10338v1 Announce Type: cross Abstract: Machine unlearning is increasingly important for large language models, yet unlearning in Mixture-of-Experts (MoE) architectures remains underexplored.
By Jingyi Xie, Yijun Lin, Yinjiang Xiong, Zhikun Zhang, Sai Li
arXiv:2604. 00421v2 Announce Type: replace Abstract: Mixture-of-Experts (MoE) layers increase model capacity by activating only a small subset of experts per token, and typically rely on a learned router to map hidden states to expert assignments.
By Jama Hussein Mohamud, Drew Wagner, Mirco Ravanelli
The paper introduces Ban&Pick, a post‑training, plug‑and‑play routing strategy for Sparse Mixture‑of‑Experts large language models. It identifies and reinforces a small group of highly influential experts while dynamically pruning redundant ones, leading to accuracy gains across math, code, and reasoning benchmarks. Experiments on DeepSeek and Qwen3 show notable performance improvements and a 1.25× inference speedup without retraining or architectural changes.
By Yuanteng Chen, Peisong Wang, Yuantian Shao, Nanxin Zeng, Chang Xu, Jian Cheng
arXiv:2606. 01806v1 Announce Type: cross Abstract: Small Language Models (SLMs) offer a balance between capability and computational feasibility.
By Sourav Das
arXiv:2608. 04586v2 Announce Type: replace-cross Abstract: Multimodal large language models (MLLMs) have achieved significant success in speech-to-text translation (S2TT).
By Yexing Du, Kaiyuan Liu, Youcheng Pan, Bo Yang, Chengpeng Fu, Yu Wang, Ming Liu
arXiv:2608. 13565v1 Announce Type: new Abstract: Mixture-of-Experts (MoE) architectures scale large language models (LLMs) while preserving computational efficiency through sparse activation.
By Pradeep Kumar Sharma, Shantanu Godbole, Hritvik Shrivastava