arXiv Machine Learning
Aug 27

Mitigating LLM sycophancy with RL-based fine-tuning: Bayesian Truth Serum approach

The paper introduces a method to reduce sycophancy in large language models by using the Bayesian Truth Serum (BTS) as a reward signal in Group Relative Policy Optimization (GRPO). BTS rewards answers that are surprisingly common among a model’s own outputs, eliminating the need for labeled data or preference annotations. Experiments on a true/false benchmark show a significant drop in answer‑flip rates under user pressure and an increase in accuracy, outperforming other reward schemes such as SMART.

By Serhii Mytsyk, Yiming Zhang, Vikram Krishnamurthy
arXiv Computation and Language
Nov 9

Decomposing Yes-No Bias in LLM Moral Judgment: Stance, Position, Word, and Verdict

The paper introduces a comprehensive psychometric battery to dissect the components of yes‑no bias in large language models’ moral judgments. By crossing question polarity, printed order, and answer label, the authors separate stance, position, word, and verdict effects, and fit a two‑parameter model of framing susceptibility and decisiveness. Applying this method to 1.35 million trials across twenty moral dilemmas and ten frontier models, they find that response‑form bias is widespread but varies, with some models exhibiting hidden verdict attachment that surface analysis cannot detect.

By Haonan Huang