arXiv Machine Learning
Sycophantic Agreement Transfers with Neutral Data via Contrastive Preference Optimization
Read the original on arXiv Machine Learning →The Flow has not summarised this story yet — read it at arXiv Machine Learning.
Related stories
arXiv Machine Learning
Aug 13
Mind the Gap: Structure-Aware Consistency in Preference Learning
arXiv:2604. 27733v2 Announce Type: replace Abstract: Aligning Large Language Models (LLMs) with human intent, whether through explicit reward modeling or direct methods such as DPO, fundamentally relies on minimizing a surrogate loss as a proxy for the true pairwise ranking objective.
arXiv AI
Aug 18
THESIS-MoE: Trainable Hierarchical Extraction and SteerIng of Sycophancy in Mixture-of-Experts
arXiv:2608. 15687v1 Announce Type: new Abstract: Sycophancy, the tendency of a language model to change its answer to match a user's stated belief, is a common alignment failure.
arXiv Computation and Language
Aug 27
Mitigating LLM biases toward spurious social contexts using direct preference optimization
The paper examines how large language models (LLMs) can be biased by irrelevant social contexts when evaluating teachers, using a large U.S. classroom transcript dataset. It shows that spurious contexts can shift model ratings by up to 1.48 points on a 7‑point scale and that standard mitigation methods like SFT and DPO are insufficient. The authors introduce Debiasing‑DPO, a method that combines contrastive reasoning‑augmented DPO with SFT, which reduces bias by 84% and improves predictive accuracy by 52% on Llama and Qwen Instruct models.
arXiv AI
Oct 5
Mitigating Social Sycophancy via Pluralistic Preference Optimization
The paper introduces Pluralistic Preference Optimization (PlurPO), a method that reduces social sycophancy in language models by having the model simulate multiple stakeholders’ perspectives when responding to interpersonal conflict scenarios. PlurPO trains the model to prefer responses acceptable to all simulated stakeholders, using only the model’s own output signals without ground‑truth labels. Experiments across four datasets and model families show significant reductions in sycophantic endorsement, including an 89% drop on harmful intent statements and halving the gap in general advice endorsement rates.
arXiv AI
Sep 10
AlignDiff: Exploiting Model-Intrinsic Information for Better Preference Data Selection
arXiv:2609.05899v1 Announce Type: cross Abstract: Aligning large language models with human preferences remains a challenge, primarily due to the critical role of preference data quality in effective...
Hugging Face Trending Papers
Jun 17
The Wrong Kind of Right: Quantifying and Localizing Misfired Alignment in LLMs
Warning: This paper studies stereotypes and biases, and contains potentially disturbing examples, used for illustration purposes only. Our findings should not be interpreted as an argument against alignment.