arXiv:2502.14643v3 Announce Type: replace Abstract: Direct Preference Optimization (DPO) is a widely adopted offline algorithm for preference-based reinforcement learning from human feedback (RLHF),...
arXiv:2606. 12505v1 Announce Type: cross Abstract: Offline preference optimization has become a practical substitute for reinforcement learning from human feedback, but pairwise objectives such as Direct Preference Optimization (DPO) and its variants use only the chosen and rejected responses stored in a static dataset.
arXiv:2509. 22851v4 Announce Type: replace-cross Abstract: Margin-based optimization is fundamental to improving generalization and robustness in classification tasks.
arXiv:2609.38860v1 Announce Type: cross Abstract: Learning from human preferences is central to large language model (LLM) alignment, but human preference annotation is costly. Active preference lear...
arXiv:2607. 25136v1 Announce Type: new Abstract: Research on preference optimization often varies the training objective while holding the data fixed.
arXiv:2604. 26360v2 Announce Type: replace-cross Abstract: Reinforcement learning from human feedback (RLHF) systems face a compounding alignment challenge: not only are learned reward models uncertain about unseen state-action pairs, but the human preference annotations they are trained on are themselves inconsistent, context-dependent, and noisy.
arXiv:2609.21525v1 Announce Type: new Abstract: Preference-based reward shaping can guide reinforcement learning, but adding preference signals to the reward may unintentionally change the task being...
arXiv:2606. 19328v1 Announce Type: cross Abstract: Preference-based RL provides an approach to learning reward models from pairwise comparisons of behaviors, bypassing the need for explicit reward design.
The paper investigates preference-based bandits where a learner selects pairs of arms and receives binary preference feedback modeled by Bradley–Terry. It introduces the locally sensitive eluder dimension, a new complexity measure for logistic preference feedback, and proposes the GINOP algorithm that uses log-loss confidence sets to balance optimism and exploration. The authors prove a first-order regret bound showing that learning with preference feedback can be as statistically efficient as learning from direct rewards, and they validate their theory with empirical experiments.
SCOPE is a framework that calibrates an acceptance threshold for large language models used as pairwise judges, ensuring that the error rate among non-abstained judgments does not exceed a user-specified level α. It introduces Bidirectional Preference Entropy (BPE) to provide a bias-neutral uncertainty signal by querying the judge in both response positions and converting the averaged preference probability into an entropy-based score. Across multiple pairwise judging benchmarks, BPE outperforms standard confidence proxies in calibration and discrimination, while SCOPE consistently meets the target risk bound (empirical FDR ≈0.097–0.099 at α=0.10) and retains substantial coverage, accepting up to 2.4× more judgments under the same risk constraint.
arXiv:2604. 18239v4 Announce Type: replace-cross Abstract: Preference optimization is widely used to align large language models (LLMs) with human preferences.
The paper introduces patterning, a reweighting technique that adjusts preference pairs based on their susceptibility to bias, to debias a Gemma 2 9B Instruct reward model trained on Skywork-Reward-Preference v0.2. Using this method, the authors achieve a +14.2 ± 1.2 percentage point improvement on the RM‑Bench Hard split while maintaining overall accuracy, matching the best reported Hard‑split gain from a comparable model. The study also demonstrates that the learned weights are interpretable, transferable across Gemma variants, and partially effective on Llama 3.1 8B.