arXiv Computation and Language

Limited Stereotype Control Through Routing Reweighting in MoE Language Models

The paper introduces Fairness-Aware Routing Equilibrium (FARE), a diagnostic framework that examines how demographic prompts are routed differently from neutral prompts in Mixture-of-Experts (MoE) language models and applies fixed inference-time reweighting to attempt stereotype control. Five MoE architectures were evaluated in English, and the reweighting procedure produced only marginal changes in stereotype preference—at most 1.3 percentage points—and did not consistently affect toxicity, lexical, or reference-overlap metrics across four generation protocols. The results indicate that the tested reweighting method offers limited effectiveness in controlling stereotypes within these models.

arXiv Computation and Language

Aug 24

RARE: Decoupling Representation Steering from Expert Routing in Mixture-of-Experts Language Models

The paper introduces RARE, a router‑agnostic representation engineering framework for Mixture‑of‑Experts language models. RARE projects behavioral perturbations onto the null space of the router matrix to avoid affecting routing, and corrects downstream routing drift. Experiments on six open‑weight MoE models show that RARE improves steering tasks—reducing harmfulness, increasing truthfulness, and enhancing factual editing—while preserving overall model accuracy.

By Zhibo Zhang, Zhen Ouyang, Ling Shi, Kailong Wang

llms benchmarks

arXiv Machine Learning

Jul 31

Fairness Pruning: Locating Demographic Bias in GLU-MLP Layers via Differential Activations

arXiv:2607. 28319v1 Announce Type: cross Abstract: This work presents Fairness Pruning, a lightweight structural intervention method designed for the management and future mitigation of demographic bias in large language models (LLMs).

By Pere Martra, Eugenio Mart\'inez C\'amara, Alfonso Ure\~na L\'opez

llms efficiency benchmarks safety

arXiv Computation and Language

Sep 17

A Probe Shift Is Not a Fairness Fix: The Limits of Representation Steering in Speech Models

arXiv:2609.18533v1 Announce Type: new Abstract: Automatic speech recognition (ASR) systems exhibit unequal error rates across speaker groups, motivating interventions on their internal representation...

By Nicolas Bourrel, Abderrahmane Issam, Gerasimos Spanakis

multimodal safety

arXiv AI

Oct 5

Frequency Is Not Sensitivity Identifying Safety-Sensitive Experts in Sparse MoE LLM

arXiv:2610.02910v1 Announce Type: new Abstract: Suppressing a small set of routed experts can weaken the safety behavior of a sparse Mixture-of-Experts (MoE) language model without retraining. Which...

By Md Nurul Absar Siddiky, Liuwan Zhu, Yingfei Dong

llms safety

arXiv AI

Sep 1

Unveiling Language Routing Isolation in Multilingual MoE Models for Interpretable Subnetwork Adaptation

arXiv:2604.03592v2 Announce Type: replace-cross Abstract: Mixture-of-Experts (MoE) models exhibit striking performance disparities across languages, yet the internal mechanisms driving these gaps rem...

By Kening Zheng, Wei-Chieh Huang, Jiahao Huo, Zhonghao Li, Henry Peng Zou, Yibo Yan, Xin Zou, Jungang Li, Junzhuo Li, Hanrong Zhang, Xuming Hu, Philip S. Yu

Hugging Face Trending Papers

Sep 16

A Probe Shift Is Not a Fairness Fix: The Limits of Representation Steering in Speech Models

Automatic speech recognition (ASR) systems exhibit unequal error rates across speaker groups, motivating interventions on their internal representations. We ask whether speaker-linked attributes that...

multimodal safety

arXiv Machine Learning

Aug 10

Let's Unlearn Stereotypes Before Decision-Making: Assessing the Impact of Intrinsic Bias Mitigation on Downstream Fairness in LLMs

arXiv:2509. 16462v2 Announce Type: replace-cross Abstract: Large Language Models (LLMs) are increasingly used in high-stakes decision-making systems, where biased predictions can reinforce social and economic disparities.

By Mina Arzaghi, Alireza Dehghanpour Farashah, Florian Carichon, Jean-Fran\c{c}ois Plante, Golnoosh Farnadi

llms rag fine-tuning benchmarks safety

arXiv AI

Oct 1

RA-MoE: Routing-Aligned Fine-Tuning for Multilingual Adaptation of Mixture-of-Experts Models

arXiv:2605.28306v2 Announce Type: replace-cross Abstract: Mixture-of-Experts (MoE) models enable efficient LLM scaling, yet adapting them to non-English downstream tasks remains challenging. Standard...

By Guanzhi Deng, Kuan Wu, Haibo Wang, Shing Yin Wong, Sichun Luo, Zengyan Liu, Fang Xu, Zhuoran Wang, Linqi Song

llms fine-tuning safety

arXiv Computation and Language

Sep 15

Intersectional Fairness in Large Language Models

arXiv:2604.20677v3 Announce Type: replace Abstract: Large Language Models (LLMs) are increasingly deployed in socially sensitive settings, raising concerns about fairness and bias, particularly when...

By Chaima Boufaied, Ronnie De Souza Santos, Ann Barcomb

llms nlp benchmarks safety

Hugging Face Trending Papers

Jul 30

Fairness Pruning: Locating Demographic Bias in GLU-MLP Layers via Differential Activations

This work presents Fairness Pruning, a lightweight structural intervention method designed for the management and future mitigation of demographic bias in large language models (LLMs). As a foundational empirical validation of this method, this work focuses on causal bias localization.

llms efficiency benchmarks safety

arXiv AI

Jun 24

Breaking the Mirror: Activation-Based Mitigation of Self-Preference in LLM Evaluators

arXiv:2509. 03647v2 Announce Type: replace-cross Abstract: Large language models (LLMs) increasingly serve as automated evaluators, yet they suffer from "self-preference bias": a tendency to favor their own outputs over those of other models.

By Dani Roytburg, Matthew Bozoukov, Matthew Nguyen, Jou Barzdukas, Simon Fu, Narmeen Oozeer

llms safety

arXiv Computation and Language

Aug 27

Lower-Resource, Higher Scores: Language Bias in LLM Evaluators

The paper demonstrates that large language model (LLM) evaluators, whether reward‑model based or prompted LLM‑as‑a‑Judge, exhibit significant language bias in multilingual settings. Experiments with semantically identical instruction‑response pairs across 23 languages reveal that lower‑resource languages receive higher scores, a bias that persists across eight open‑weight evaluators and is not detectable by standard pairwise accuracy metrics. The authors link the bias to model uncertainty and language identity, showing it cannot be explained by content difficulty alone.

By Ej Zhou, Lucas Resck, Zheng Hui, Anna Korhonen

llms safety

Related stories