The study examines whether Mixture-of-Experts (MoE) language models develop linguistically structured expert routing during bilingual language acquisition. Using a decoder-only English‑German MoE Transformer trained with sequential language exposure, the authors probe token‑level routing distributions and measure category‑dependent specialization via mutual information, routing entropy, and Jensen‑Shannon distance. Results show that a curriculum‑trained model peaks at a mutual information of 0.1148 at layer 5, while a no‑curriculum baseline trained on mixed data achieves a higher peak of 0.2599 at the same layer, indicating stronger aggregate specialization. Replication with a second training seed reveals that the no‑curriculum condition’s specialization focuses on a single language in a seed‑dependent way, whereas the curriculum consistently yields a stable, language‑balanced routing profile, suggesting that staged bilingual exposure reduces single‑language dominance.
By Amrit Gopinath, Raghul, Durairaj Thenmozhi
arXiv:2604.03592v2 Announce Type: replace-cross
Abstract: Mixture-of-Experts (MoE) models exhibit striking performance disparities across languages, yet the internal mechanisms driving these gaps rem...
By Kening Zheng, Wei-Chieh Huang, Jiahao Huo, Zhonghao Li, Henry Peng Zou, Yibo Yan, Xin Zou, Jungang Li, Junzhuo Li, Hanrong Zhang, Xuming Hu, Philip S. Yu
arXiv:2605.28306v2 Announce Type: replace-cross
Abstract: Mixture-of-Experts (MoE) models enable efficient LLM scaling, yet adapting them to non-English downstream tasks remains challenging. Standard...
By Guanzhi Deng, Kuan Wu, Haibo Wang, Shing Yin Wong, Sichun Luo, Zengyan Liu, Fang Xu, Zhuoran Wang, Linqi Song
arXiv:2605.16026v2 Announce Type: replace-cross
Abstract: Multilingual speech translation frontends built on speech large language models typically condition on language identity, as a learned embedd...
By Yu Pan, Tianjiao Zhang, Yang Hou, Xiongfei Wu, Liang Zhang, Yves Le Traon, Lei Ma, Jianjun Zhao
ScoutRoute is a lightweight post‑hoc routing framework for Mixture‑of‑Experts systems that predicts expert competence using internal language model signals from a single forward pass of a scout model. It employs hidden‑state representations with logistic probes and logit‑derived confidence measures in a domain‑aware routing strategy, avoiding modifications to the underlying language models. Experiments on five open‑weight language models across four benchmark datasets show that ScoutRoute consistently outperforms individual routing signals and achieves superior calibration with only one scout‑model inference.
By Rajdeep Pandhare, Delzaan Bharucha
The paper introduces RARE, a router‑agnostic representation engineering framework for Mixture‑of‑Experts language models. RARE projects behavioral perturbations onto the null space of the router matrix to avoid affecting routing, and corrects downstream routing drift. Experiments on six open‑weight MoE models show that RARE improves steering tasks—reducing harmfulness, increasing truthfulness, and enhancing factual editing—while preserving overall model accuracy.
By Zhibo Zhang, Zhen Ouyang, Ling Shi, Kailong Wang
arXiv:2606. 14398v1 Announce Type: new Abstract: Mixture-of-experts (MoE) layers enable the scaling of transformer models while keeping the inference compute fixed.
By Yongli Xiang, Vinoth Nandakumar, Yunzhi Yao, Peike Li, Tongliang Liu
The paper introduces two token-error supervision methods for sparse mixture-of-experts (MoE) large language models, aligning routing affinities with token-level cross-entropy loss. The first method predicts an error score per expert and uses it to adjust affinities before top‑K selection, while the second directly aligns router affinities to the model’s objective without extra heads or inference changes. Experiments on Granite and ARC‑Challenge show accuracy gains of about 2.3–2.94 percentage points over a parameter‑matched baseline, preserving the native sparse execution budget.
By Yury Nahshan, Nati Daniel, Jacob Goldberger, Yoli Shavit
arXiv:2610.23249v1 Announce Type: new
Abstract: Small-to-large language model routing can reduce inference cost by escalating difficult inputs while retaining easier ones on a smaller model. Its succ...
By Md Sabbir Hossen, Anichur Rahman, Pabon Shaha, Airin Afroj Aishi, Andrew H. Sung, Md Shohel Rana
arXiv:2608. 06396v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) language models route each token through a small subset of experts, making routing patterns useful for identifying task-relevant experts during downstream adaptation.
By Guanzhi Deng, Haibo Wang, Kuan Wu, Xiangru Jian, Shing Yin Wong, Sichun Luo, Zhuoran Wang, Linqi Song
The paper investigates whether different latent language probes—GMM-based representation probes and decoding-based probes—measure the same phenomenon in multilingual language models. Across various model families, training regimes, domains, tasks, checkpoints, and up to 27 languages, the authors find systematic disagreement: representation probes indicate earlier cross‑lingual mixing, while decoding probes reveal sharper, English‑biased language signals. These differences correlate with model multilinguality and training progression but remain relatively stable across domains, suggesting that current probes capture distinct aspects of multilingual processing rather than a single internal lingua franca.
By Deniz Bayazit, Badr AlKhamissi, Antoine Bosselut
The paper demonstrates that large language model (LLM) evaluators, whether reward‑model based or prompted LLM‑as‑a‑Judge, exhibit significant language bias in multilingual settings. Experiments with semantically identical instruction‑response pairs across 23 languages reveal that lower‑resource languages receive higher scores, a bias that persists across eight open‑weight evaluators and is not detectable by standard pairwise accuracy metrics. The authors link the bias to model uncertainty and language identity, showing it cannot be explained by content difficulty alone.
By Ej Zhou, Lucas Resck, Zheng Hui, Anna Korhonen