arXiv Computation and Language October 9, 2026 By Junhyeok Lee, Han Jang, Kyu Sung Choi

Limited Stereotype Control Through Routing Reweighting in MoE Language Models

Read the original on arXiv Computation and Language →

The paper introduces Fairness-Aware Routing Equilibrium (FARE), a diagnostic framework that examines how demographic prompts are routed differently from neutral prompts in Mixture-of-Experts (MoE) language models and applies fixed inference-time reweighting to attempt stereotype control. Five MoE architectures were evaluated in English, and the reweighting procedure produced only marginal changes in stereotype preference—at most 1.3 percentage points—and did not consistently affect toxicity, lexical, or reference-overlap metrics across four generation protocols. The results indicate that the tested reweighting method offers limited effectiveness in controlling stereotypes within these models.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

arXiv Computation and Language

Aug 24

RARE: Decoupling Representation Steering from Expert Routing in Mixture-of-Experts Language Models

The paper introduces RARE, a router‑agnostic representation engineering framework for Mixture‑of‑Experts language models. RARE projects behavioral perturbations onto the null space of the router matrix to avoid affecting routing, and corrects downstream routing drift. Experiments on six open‑weight MoE models show that RARE improves steering tasks—reducing harmfulness, increasing truthfulness, and enhancing factual editing—while preserving overall model accuracy.

By Zhibo Zhang, Zhen Ouyang, Ling Shi, Kailong Wang

llms benchmarks

arXiv Machine Learning

Jul 31

Fairness Pruning: Locating Demographic Bias in GLU-MLP Layers via Differential Activations

arXiv:2607. 28319v1 Announce Type: cross Abstract: This work presents Fairness Pruning, a lightweight structural intervention method designed for the management and future mitigation of demographic bias in large language models (LLMs).

By Pere Martra, Eugenio Mart\'inez C\'amara, Alfonso Ure\~na L\'opez

llms efficiency benchmarks safety

arXiv Computation and Language

Sep 17

A Probe Shift Is Not a Fairness Fix: The Limits of Representation Steering in Speech Models

arXiv:2609.18533v1 Announce Type: new Abstract: Automatic speech recognition (ASR) systems exhibit unequal error rates across speaker groups, motivating interventions on their internal representation...

By Nicolas Bourrel, Abderrahmane Issam, Gerasimos Spanakis

multimodal safety

arXiv AI

Oct 5

Frequency Is Not Sensitivity Identifying Safety-Sensitive Experts in Sparse MoE LLM

arXiv:2610.02910v1 Announce Type: new Abstract: Suppressing a small set of routed experts can weaken the safety behavior of a sparse Mixture-of-Experts (MoE) language model without retraining. Which...

By Md Nurul Absar Siddiky, Liuwan Zhu, Yingfei Dong

llms safety

arXiv AI

Sep 1

Unveiling Language Routing Isolation in Multilingual MoE Models for Interpretable Subnetwork Adaptation

arXiv:2604.03592v2 Announce Type: replace-cross Abstract: Mixture-of-Experts (MoE) models exhibit striking performance disparities across languages, yet the internal mechanisms driving these gaps rem...

By Kening Zheng, Wei-Chieh Huang, Jiahao Huo, Zhonghao Li, Henry Peng Zou, Yibo Yan, Xin Zou, Jungang Li, Junzhuo Li, Hanrong Zhang, Xuming Hu, Philip S. Yu

Hugging Face Trending Papers

Sep 16

A Probe Shift Is Not a Fairness Fix: The Limits of Representation Steering in Speech Models

Automatic speech recognition (ASR) systems exhibit unequal error rates across speaker groups, motivating interventions on their internal representations. We ask whether speaker-linked attributes that...

multimodal safety

One email a morning, machine-written

Related stories

RARE: Decoupling Representation Steering from Expert Routing in Mixture-of-Experts Language Models

Fairness Pruning: Locating Demographic Bias in GLU-MLP Layers via Differential Activations

A Probe Shift Is Not a Fairness Fix: The Limits of Representation Steering in Speech Models

Frequency Is Not Sensitivity Identifying Safety-Sensitive Experts in Sparse MoE LLM

Unveiling Language Routing Isolation in Multilingual MoE Models for Interpretable Subnetwork Adaptation

A Probe Shift Is Not a Fairness Fix: The Limits of Representation Steering in Speech Models