The Flow has not summarised this story yet — read it at Hugging Face Trending Papers.
The study examines whether Mixture-of-Experts (MoE) language models develop linguistically structured expert routing during bilingual language acquisition. Using a decoder-only English‑German MoE Transformer trained with sequential language exposure, the authors probe token‑level routing distributions and measure category‑dependent specialization via mutual information, routing entropy, and Jensen‑Shannon distance. Results show that a curriculum‑trained model peaks at a mutual information of 0.1148 at layer 5, while a no‑curriculum baseline trained on mixed data achieves a higher peak of 0.2599 at the same layer, indicating stronger aggregate specialization. Replication with a second training seed reveals that the no‑curriculum condition’s specialization focuses on a single language in a seed‑dependent way, whereas the curriculum consistently yields a stable, language‑balanced routing profile, suggesting that staged bilingual exposure reduces single‑language dominance.
arXiv:2604.03592v2 Announce Type: replace-cross Abstract: Mixture-of-Experts (MoE) models exhibit striking performance disparities across languages, yet the internal mechanisms driving these gaps rem...
The paper introduces RARE, a router‑agnostic representation engineering framework for Mixture‑of‑Experts language models. RARE projects behavioral perturbations onto the null space of the router matrix to avoid affecting routing, and corrects downstream routing drift. Experiments on six open‑weight MoE models show that RARE improves steering tasks—reducing harmfulness, increasing truthfulness, and enhancing factual editing—while preserving overall model accuracy.
arXiv:2606. 14398v1 Announce Type: new Abstract: Mixture-of-experts (MoE) layers enable the scaling of transformer models while keeping the inference compute fixed.
arXiv:2608. 06396v1 Announce Type: cross Abstract: Mixture-of-Experts (MoE) language models route each token through a small subset of experts, making routing patterns useful for identifying task-relevant experts during downstream adaptation.
The paper investigates whether different latent language probes—GMM-based representation probes and decoding-based probes—measure the same phenomenon in multilingual language models. Across various model families, training regimes, domains, tasks, checkpoints, and up to 27 languages, the authors find systematic disagreement: representation probes indicate earlier cross‑lingual mixing, while decoding probes reveal sharper, English‑biased language signals. These differences correlate with model multilinguality and training progression but remain relatively stable across domains, suggesting that current probes capture distinct aspects of multilingual processing rather than a single internal lingua franca.