← Back to all news
arXiv AI By Sekh Mainul Islam, Pepa Atanasova, Isabelle Augenstein

Explaining the Roles of Experts in Mixture-of-Experts LLMs for Personalization

Read the original on arXiv AI →

The paper investigates how to personalize Large Language Models by leveraging Mixture-of-Experts (MoE) architectures, which route tokens to a few specialized experts, thereby reducing polysemanticity compared to dense models. The authors adapt three open-weight MoE LLMs on a large personalization dataset, train crosscoders to learn interpretable personalized latent representations, and identify experts that encode distinct personalization features. Experiments show that personalized MoE LLMs match dense LLM performance while activating fewer parameters, exhibit lower polysemanticity, and that causal interventions on the identified experts can steer a non-personalized model toward personalized behavior.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
2d ago

Interpretability Is Not Just a Tax: Sparsity and Diversity Yield Legible and Composable Reward Experts for Personalization

The paper investigates how to make Mixture-of-Experts (MoE) reward models more interpretable and useful for personalization by focusing on sparsity and diversity regularizers. By keeping the architecture and training data constant, the authors show that conventional MoE reward models learn entangled components that hinder both interpretability and personalization. Introducing a sparse MoE objective reduces entanglement, yielding experts that align with semantic categories and functional strengths, and enables composable, few‑shot personalization with a 25.81‑point improvement over entangled models.

By Yifan Wang, Jinyi Mu, Mayank Jobanputra, Yu Wang, Ji-Ung Lee, Soyoung Oh, Isabel Valera, Vera Demberg
efficiencysafety
More like this →
arXiv Computation and Language
Sep 10

HyperTrace: Hypothesis-Based Preference Tracing for Online LLM Personalization

HyperTrace is a training‑free framework that personalizes large language models by tracing latent user preferences online. It maintains interpretable natural‑language hypotheses about short‑term intent and long‑term preferences, updating them with an SMC‑style reweighting process driven by an LLM‑based surrogate choice model. Experiments on PRISM and PersonaMem‑v2 demonstrate that HyperTrace improves response alignment, preference prediction, and profile consistency compared to strong online baselines.

By Jianzhi Shen, Keyu Mao, Minghao Shao, Chuanyang Jin, Yusong Wang, Ailiang Lin, Kotaro Funakoshi, Manabu Okumura, Tianmin Shu, Muhammad Shafique
llmssafety
More like this →
arXiv Computation and Language
Sep 3

Beyond Retrieval: Learning Compact User Representations for Scalable LLM Personalization

The paper introduces TAP-PER, a prefix‑based framework that learns compact user representations for large language model personalization. By encoding user preferences into lightweight prefix embeddings and incorporating temporal signals, TAP‑PER avoids the need for heavy per‑user adapters or prompt‑serialized histories. Experiments on six LaMP tasks show that TAP‑PER outperforms both prompt‑based and model‑based baselines while using far fewer per‑user parameters, enabling scalable personalization at large user scales.

By Heng Cao, Fan Zhang, Jian Yao, Yujie Zheng, Changlin Zhao, Lu Hao, Yuxuan Wei, Wangze Ni, Huaiyu Fu, Yuqian Sun, Xuyan Mo
llmsrag
More like this →