arXiv AI
Unveiling Language Routing Isolation in Multilingual MoE Models for Interpretable Subnetwork Adaptation
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv Computation and Language
4d ago
A Declarative-Procedural Perspective on Expert Routing in Bilingual Mixture-of-Experts Language Models
The study examines whether Mixture-of-Experts (MoE) language models develop linguistically structured expert routing during bilingual language acquisition. Using a decoder-only English‑German MoE Transformer trained with sequential language exposure, the authors probe token‑level routing distributions and measure category‑dependent specialization via mutual information, routing entropy, and Jensen‑Shannon distance. Results show that a curriculum‑trained model peaks at a mutual information of 0.1148 at layer 5, while a no‑curriculum baseline trained on mixed data achieves a higher peak of 0.2599 at the same layer, indicating stronger aggregate specialization. Replication with a second training seed reveals that the no‑curriculum condition’s specialization focuses on a single language in a seed‑dependent way, whereas the curriculum consistently yields a stable, language‑balanced routing profile, suggesting that staged bilingual exposure reduces single‑language dominance.
Hugging Face Trending Papers
Aug 15
A Declarative-Procedural Perspective on Expert Routing in Bilingual Mixture-of-Experts Language Models
We investigate whether Mixture-of-Experts (MoE) language models develop linguistically structured expert routing during bilingual language acquisition. Inspired by the Declarative-Procedural framework...
arXiv Computation and Language
Aug 24
RARE: Decoupling Representation Steering from Expert Routing in Mixture-of-Experts Language Models
The paper introduces RARE, a router‑agnostic representation engineering framework for Mixture‑of‑Experts language models. RARE projects behavioral perturbations onto the null space of the router matrix to avoid affecting routing, and corrects downstream routing drift. Experiments on six open‑weight MoE models show that RARE improves steering tasks—reducing harmfulness, increasing truthfulness, and enhancing factual editing—while preserving overall model accuracy.
arXiv AI
Aug 10
TEXAS: Task-Expert-Aware Supervision for Downstream Mixture-of-Experts LLM Adaptation
arXiv:2608. 06396v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) language models route each token through a small subset of experts, making routing patterns useful for identifying task-relevant experts during downstream adaptation.
arXiv AI
Jul 28
LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in Large Language Models
arXiv:2411. 00918v5 Announce Type: replace-cross Abstract: Mixture of experts (MoE) architectures have become a cornerstone for scaling up and are a key component in most large language models such as GPT-OSS, DeepSeek-V3, Llama-4, and Gemini-2.
arXiv AI
Jun 16
SPRI: SVD-Partitioned Residual Initialization for Data-Constrained MoE Upcycling
arXiv:2606. 16456v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) models enable efficient scaling, but training them from scratch remains prohibitively expensive.