arXiv Computation and Language August 31, 2026 By Amrit Gopinath, Raghul, Durairaj Thenmozhi

A Declarative-Procedural Perspective on Expert Routing in Bilingual Mixture-of-Experts Language Models

Read the original on arXiv Computation and Language →

The study examines whether Mixture-of-Experts (MoE) language models develop linguistically structured expert routing during bilingual language acquisition. Using a decoder-only English‑German MoE Transformer trained with sequential language exposure, the authors probe token‑level routing distributions and measure category‑dependent specialization via mutual information, routing entropy, and Jensen‑Shannon distance. Results show that a curriculum‑trained model peaks at a mutual information of 0.1148 at layer 5, while a no‑curriculum baseline trained on mixed data achieves a higher peak of 0.2599 at the same layer, indicating stronger aggregate specialization. Replication with a second training seed reveals that the no‑curriculum condition’s specialization focuses on a single language in a seed‑dependent way, whereas the curriculum consistently yields a stable, language‑balanced routing profile, suggesting that staged bilingual exposure reduces single‑language dominance.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Hugging Face Trending Papers

Aug 15

A Declarative-Procedural Perspective on Expert Routing in Bilingual Mixture-of-Experts Language Models

We investigate whether Mixture-of-Experts (MoE) language models develop linguistically structured expert routing during bilingual language acquisition. Inspired by the Declarative-Procedural framework...

llms safety

arXiv AI

Oct 1

RA-MoE: Routing-Aligned Fine-Tuning for Multilingual Adaptation of Mixture-of-Experts Models

arXiv:2605.28306v2 Announce Type: replace-cross Abstract: Mixture-of-Experts (MoE) models enable efficient LLM scaling, yet adapting them to non-English downstream tasks remains challenging. Standard...

By Guanzhi Deng, Kuan Wu, Haibo Wang, Shing Yin Wong, Sichun Luo, Zengyan Liu, Fang Xu, Zhuoran Wang, Linqi Song

llms fine-tuning safety

arXiv AI

Sep 1

Unveiling Language Routing Isolation in Multilingual MoE Models for Interpretable Subnetwork Adaptation

arXiv:2604.03592v2 Announce Type: replace-cross Abstract: Mixture-of-Experts (MoE) models exhibit striking performance disparities across languages, yet the internal mechanisms driving these gaps rem...

By Kening Zheng, Wei-Chieh Huang, Jiahao Huo, Zhonghao Li, Henry Peng Zou, Yibo Yan, Xin Zou, Jungang Li, Junzhuo Li, Hanrong Zhang, Xuming Hu, Philip S. Yu

arXiv AI

4d ago

Conditioned in Training, Dispensable at Inference: Typological Conditioning for Multilingual Speech Translation

arXiv:2605.16026v2 Announce Type: replace-cross Abstract: Multilingual speech translation frontends built on speech large language models typically condition on language identity, as a learned embedd...

By Yu Pan, Tianjiao Zhang, Yang Hou, Xiongfei Wu, Liang Zhang, Yves Le Traon, Lei Ma, Jianjun Zhao

llms rag benchmarks

arXiv AI

22h ago

ScoutRoute: Efficient Mixture-of-Experts Routing Using Internal Language Model Signals

ScoutRoute is a lightweight post‑hoc routing framework for Mixture‑of‑Experts systems that predicts expert competence using internal language model signals from a single forward pass of a scout model. It employs hidden‑state representations with logistic probes and logit‑derived confidence measures in a domain‑aware routing strategy, avoiding modifications to the underlying language models. Experiments on five open‑weight language models across four benchmark datasets show that ScoutRoute consistently outperforms individual routing signals and achieves superior calibration with only one scout‑model inference.

By Rajdeep Pandhare, Delzaan Bharucha

llms

arXiv Computation and Language

22h ago

Who Pays for Efficient Inference? Confidence-Based Small-to-Large LLM Routing Across Tasks and Languages

arXiv:2610.23249v1 Announce Type: new Abstract: Small-to-large language model routing can reduce inference cost by escalating difficult inputs while retaining easier ones on a smaller model. Its succ...

By Md Sabbir Hossen, Anichur Rahman, Pabon Shaha, Airin Afroj Aishi, Andrew H. Sung, Md Shohel Rana

llms benchmarks

One email a morning, machine-written

Related stories

A Declarative-Procedural Perspective on Expert Routing in Bilingual Mixture-of-Experts Language Models

RA-MoE: Routing-Aligned Fine-Tuning for Multilingual Adaptation of Mixture-of-Experts Models

Unveiling Language Routing Isolation in Multilingual MoE Models for Interpretable Subnetwork Adaptation

Conditioned in Training, Dispensable at Inference: Typological Conditioning for Multilingual Speech Translation

ScoutRoute: Efficient Mixture-of-Experts Routing Using Internal Language Model Signals

Who Pays for Efficient Inference? Confidence-Based Small-to-Large LLM Routing Across Tasks and Languages