arXiv:2606. 04284v1 Announce Type: cross Abstract: Preference modeling plays a central role in reinforcement learning from human feedback (RLHF), enabling large language models (LLMs) to align with human values.
By Yifan Wang, Jinyi Mu, Mayank Jobanputra, Yu Wang, Ji-Ung Lee, Soyoung Oh, Isabel Valera, Vera Demberg
arXiv:2608. 06400v1 Announce Type: new Abstract: Reward models are central to learning from human preferences, yet identifying what drives their predictions remains challenging.
By Yifan Wang, Jinyi Mu, Mayank Jobanputra, Yu Wang, Soyoung Oh, Isabel Valera, Vera Demberg
arXiv:2606. 12360v1 Announce Type: new Abstract: Language-model post-training is the main stage at which model behavior is shaped, yet it still largely involves optimization of scalar rewards that summarize diverse desiderata.
By Leon Bergen, Usha Bhalla, Sidharth Baskaran, Max Loeffler, Raphael Sarfati, Dhruvil Gala, Ryan Panwar, Santiago Aranguri, Thomas Fel, Atticus Geiger, Matthew Kowal, Siddharth Boppana, Daniel Balsam, Owen Lewis, Jack Merullo, Thomas McGrath, Ekdeep Singh Lubana
Language-model post-training is the main stage at which model behavior is shaped, yet it still largely involves optimization of scalar rewards that summarize diverse desiderata. This abstraction gives practitioners little visibility into what their data actually teaches models, allowing spurious correlations to be learned by a model and inducing undesirable behaviors such as over-stylization and sycophancy.
The paper investigates how to personalize Large Language Models by leveraging Mixture-of-Experts (MoE) architectures, which route tokens to a few specialized experts, thereby reducing polysemanticity compared to dense models. The authors adapt three open-weight MoE LLMs on a large personalization dataset, train crosscoders to learn interpretable personalized latent representations, and identify experts that encode distinct personalization features. Experiments show that personalized MoE LLMs match dense LLM performance while activating fewer parameters, exhibit lower polysemanticity, and that causal interventions on the identified experts can steer a non-personalized model toward personalized behavior.
By Sekh Mainul Islam, Pepa Atanasova, Isabelle Augenstein
The paper investigates federated learning where each client’s data consists of unknown mixtures of distinct tasks, a scenario termed compound heterogeneity. It shows that when tasks share a common feature geometry, the optimal model for a mixed client is a convex combination of task‑specific models, motivating input‑dependent routing to specialized experts. The authors propose FedSEE, a method that recovers task experts via a convex program and achieves better performance than baselines, reducing negative transfer by 2.9 points overall and 3.7 points for the worst‑served quartile.
By Hojat Allah Salehi, Mehrdad Mahdavi, Andrew Arash Mahyari, M. Hadi Amini