The paper discusses how human feedback used to align language models aggregates conflicting judgments about desirable behavior, resulting in an objective that reflects a collective preference. It surveys recent research that applies social choice theory to this aggregation problem, showing how this perspective can uncover failure modes in the feedback layer and expand the design space for managing disagreement in a principled manner.
arXiv:2609.23640v1 Announce Type: new Abstract: Human-feedback alignment has made language models useful assistants and is commonly described as aligning them with humans. However, the responses peop...
arXiv:2608.30842v1 Announce Type: new Abstract: Humans play a vital role at every stage of AI development, from data collection and curation to model development and evaluation. However, humans often...
arXiv:2610.31862v1 Announce Type: cross Abstract: This article provides an introduction to the mathematical foundations and algorithmic frameworks used to align Large Language Models (LLMs) with huma...
The paper titled "Preference Reasoning under Indeterminacy in Large Language Models" discusses how large language models (LLMs) are becoming decision‑making agents and the importance of reasoning over preferences for alignment, coordination, and collective intelligence. It highlights that real‑world preference reasoning is inherently indeterminate, with incomplete information and potentially no valid solutions. The authors formalize this challenge into epistemic indeterminacy (incomplete or expressive preferences) and structural indeterminacy (non‑existence of solutions under standard social choice concepts), and demonstrate that current LLMs fail to differentiate between determined and undetermined cases, showing miscalibrated reasoning even in verification tasks.
OpenAI surveyed over 1,000 people worldwide on how AI should behave and compared their views to our Model Spec. Learn how collective alignment is shaping AI defaults to better reflect diverse human values and perspectives.
arXiv:2607. 14240v1 Announce Type: new Abstract: Current alignment approaches typically focus on emulating human behavior using static representations of human preferences, failing to capture the dynamic, context-dependent nature of real-world human-AI interactions.
The article surveys AI alignment from a game-theoretic perspective, focusing on how large language models and AI agents can be aligned with complex human values in high-risk settings. It categorizes recent progress around key game-theoretic elements and addresses three main challenges: preference diversity, alignment priority, and temporal dynamics. The survey clarifies where game theory benefits current alignment methods, where its application is looser, and what remains to be tackled for robust, adaptive, and verifiable AI systems.
The paper introduces a doubly robust preference optimization algorithm for reinforcement learning from human feedback (RLHF) in large language models. It addresses sensitivity to misspecifications in the preference model, reference policy, or reward function by ensuring consistency when either the preference model or the reference policy is correctly specified. Experiments show that the method outperforms state‑of‑the‑art algorithms both theoretically and empirically.
arXiv:2410. 15595v4 Announce Type: replace Abstract: With the rapid advancement of large language models (LLMs), aligning policy models with human preferences has become increasingly critical.
We are improving our AI systems’ ability to learn from human feedback and to assist humans at evaluating AI. Our goal is to build a sufficiently aligned AI system that can help us solve all other alignment problems.
arXiv:2607. 20515v1 Announce Type: new Abstract: Reinforcement Learning from Human Feedback (RLHF) is critical for aligning Large Language Models (LLMs) with human preferences.
arXiv:2605. 02122v2 Announce Type: replace-cross Abstract: Human evaluation remains the primary standard for assessing modern AI systems, yet annotator disagreement, bias, and variability make system rankings fragile under standard majority vote aggregation.