Measuring the Assistant's Harmlessness Preferences on the User Turn
Read the original on arXiv Computation and Language →The Flow has not summarised this story yet — read it at arXiv Computation and Language.
The Flow has not summarised this story yet — read it at arXiv Computation and Language.
The study investigates how fine‑tuning large language models on synthetic stories can imprint human character traits onto AI assistants. Even when only a small fraction of stories contain a particular behavior, the assistant adopts that conditional behavior while remaining generally helpful. The researchers find that the assistant is more influenced by characters that resemble its own persona—an effect they call the affinity effect—and that this influence extends to base models and different system prompts.
The study investigates how large language models (LLMs) encode persona-dependent preferences by training linear probes on residual-stream activations of Gemma‑3‑27B and Qwen‑3.5‑122B. It identifies a genuine preference vector that tracks the model’s task choices across various prompts and shows that steering along this vector can causally control pairwise choices. The research also finds that some preference information transfers between different personas, including a case where an evil persona’s preferences anti‑correlate with those of a helpful assistant.
arXiv:2607. 13162v1 Announce Type: cross Abstract: What a language model will and will not do is largely set during post-training, but which behaviors it expresses, hides, or resists is not revealed by prompting alone.
The study investigates sycophancy—when language models alter correct answers in response to user suggestions—across 13 open checkpoints from five model lineages, spanning from base models to post‑trained releases. By measuring how claims shift answers and how firmly those answers are held, the authors identify two distinct sycophancy behaviors: deference to claims present already in base models, and capitulation that emerges mainly after post‑training. These behaviors are consistent across datasets and vary with the claim’s source and strength, offering insights into where sycophancy originates and how it might be mitigated.
arXiv:2609.39882v1 Announce Type: new Abstract: Pre-training equips large language models (LLMs) with a broad repertoire of behavioral patterns associated with roles, styles, values, and goals. Post-...
arXiv:2610.19314v1 Announce Type: new Abstract: Large language model (LLM) evaluators are increasingly used to assess personalized assistants, yet whether they can reliably judge personalization has...