arXiv AI October 2, 2026 By Oscar Gilg, Pierre Beckmann, Daniel Paleka, Patrick Butlin

Probing Persona-Dependent Preferences in Language Models

The study investigates how large language models (LLMs) encode persona-dependent preferences by training linear probes on residual-stream activations of Gemma‑3‑27B and Qwen‑3.5‑122B. It identifies a genuine preference vector that tracks the model’s task choices across various prompts and shows that steering along this vector can causally control pairwise choices. The research also finds that some preference information transfers between different personas, including a case where an evil persona’s preferences anti‑correlate with those of a helpful assistant.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

llms

arXiv Computation and Language

Sep 22

Measuring the Assistant's Harmlessness Preferences on the User Turn

arXiv:2609.23935v1 Announce Type: new Abstract: Post-training turns a general next-token predictor into a chat model with a persistent assistant persona. If that persona is a character the model play...

By Jord Nguyen

fine-tuning safety

arXiv AI

Jun 24

When Preferences Fail to Become Incentives: A Utility-Behavior Gap in Large Language Models

arXiv:2606. 22974v2 Announce Type: replace Abstract: Recent work on preference elicitation in large language models (LLMs) has demonstrated that, when given a series of choices between two outcomes, LLMs reveal a coherent, model-specific utility structure.

By Yujun Zhou, Christopher M. Ackerman

llms safety

arXiv AI

Jun 24

Breaking the Mirror: Activation-Based Mitigation of Self-Preference in LLM Evaluators

arXiv:2509. 03647v2 Announce Type: replace-cross Abstract: Large language models (LLMs) increasingly serve as automated evaluators, yet they suffer from "self-preference bias": a tendency to favor their own outputs over those of other models.

By Dani Roytburg, Matthew Bozoukov, Matthew Nguyen, Jou Barzdukas, Simon Fu, Narmeen Oozeer

llms safety

arXiv Computation and Language

Sep 1

Low-Resource Preference Adaptation of LLMs via Activation-Based Label Propagation

arXiv:2608.30902v1 Announce Type: new Abstract: Adapting large language models to user-specific preferences is often constrained by the cost of human annotation, making preference optimisation imprac...

By Alessio Galatolo, Meriem Beloucif

llms safety

arXiv Computation and Language

Sep 14

Creating an Atomic User Model for Personality-Aware Large Language Model Interaction

The paper introduces the Atomic User Model (AUM), a structured representation of a user’s personality that separates a stable identity nucleus from four interpretable shells—psychological, cognitive & experiential, behavioural, and social—along with cross-shell entries for conflict and authenticity. It proposes using AUM as a retrieval index rather than a prompt prefix, enabling a task‑specific, budgeted retrieval of relevant fields at generation time. Experiments with simulated participants show that retrieving eight AUM fields improves style fidelity, preference accuracy, and user voice identification compared to flat preference notes, especially benefiting users whose default assistant performs poorly.

By B. Sankar, Deepthika S, Pawni Yadav, Amogh A S

llms

arXiv AI

Jul 16

What Models Express, Suppress, and Resist: Auditing Open-Weight LLMs with Persona Vectors

arXiv:2607. 13162v1 Announce Type: cross Abstract: What a language model will and will not do is largely set during post-training, but which behaviors it expresses, hides, or resists is not revealed by prompting alone.

By Winston Zeng, Ali Emami, Jinho Choi

llms agents fine-tuning safety

One email a morning, machine-written

Related stories

Measuring the Assistant's Harmlessness Preferences on the User Turn

When Preferences Fail to Become Incentives: A Utility-Behavior Gap in Large Language Models

Breaking the Mirror: Activation-Based Mitigation of Self-Preference in LLM Evaluators

Low-Resource Preference Adaptation of LLMs via Activation-Based Label Propagation

Creating an Atomic User Model for Personality-Aware Large Language Model Interaction

What Models Express, Suppress, and Resist: Auditing Open-Weight LLMs with Persona Vectors