← Back to all news
arXiv AI

Beyond Pairs: Your Language Model is Secretly Optimizing a Preference Graph

The paper introduces Graph Direct Preference Optimization (GraphDPO), a generalization of Direct Preference Optimization that uses directed acyclic graphs to capture richer preference structures from multiple rollouts per prompt. GraphDPO encodes dominance relations as edges and optimizes a Plackett–Luce-inspired objective that aggregates supervision over dominated neighborhoods, preserving transitivity and handling ties or incomparable responses via an equivalence-class construction. Experiments on reasoning and program synthesis tasks show that this graph-structured approach outperforms traditional pairwise and listwise alignment objectives while maintaining linear per‑prompt complexity.

Related stories

arXiv Machine Learning
Oct 8

Continuous-Utility Direct Preference Optimization

Continuous-Utility Direct Preference Optimization (CU‑DPO) replaces binary preference labels with continuous scores to better capture fine‑grained reasoning quality. The framework aligns language models to multiple prompt‑based cognitive strategies, proving a θ(K log K) sample‑complexity improvement and convergence to an entropy‑regularized utility‑maximizing policy. A two‑stage pipeline—strategy selection via best‑vs‑all comparisons and execution refinement using margin‑stratified pairs—yields significant gains on mathematical reasoning, code generation, and causal reasoning benchmarks.

By Muhammad Ahmed Mohsin, Muhammad Umer, Emily Fox
llmsreinforcement-learningbenchmarks
More like this →
arXiv AI
Oct 2

Gradient-Aligned Pair Selection for Personalized Preference Optimization

The paper introduces GAP-DPO, a method for personalizing large language models by selecting preference pairs based on gradient alignment with user utility. It formalizes personalized preference learning as a geometry‑aligned optimization problem, showing that off‑policy sampling can shift DPO updates from error correction to reinforcement when preference margins align with utility gradients. Experiments demonstrate that GAP‑DPO improves stylistic fidelity, preference alignment, and overall generation quality over standard DPO variants.

By Ruoming Jin, Xinyu Li, Hao Zhou, Jianfeng Zhu, Ruixin Guo, Feodor Dragan, Lei Xu, Haixun Wang, Yang Zhou
llmsreinforcement-learningbenchmarkssafety
More like this →
arXiv Computation and Language
Aug 28

Instruction Quality Matters: Refining Instructions for Effective Preference Learning

The paper investigates how the quality of instructions used to generate response pairs affects preference learning for language models. It shows that low‑quality or ambiguous instructions limit the range of response quality, weakening preference signals, and introduces an instruction‑refinement pipeline that improves data quality without discarding examples. Experiments across models and benchmarks demonstrate that refining instructions leads to better alignment and complements other data‑improvement methods.

By Seohyeong Lee, Hwaran Lee, Buru Chang
llmsbenchmarkssafety
More like this →
arXiv Computation and Language
Oct 5

GAW-PO: Preference Optimization with Gradient-Aligned Token Weights

The paper introduces GAW-PO, a gradient‑aligned token reweighting technique for Direct Preference Optimization (DPO). It assigns weaker penalties to rejected tokens whose gradients align with preferred behavior, while maintaining stronger penalties for conflicting tokens. Across 11 benchmarks in mathematics, reasoning, coding, and question answering, GAW‑PO outperforms standard DPO and other baselines, and remains robust when the DPO regularization parameter is reduced.

By Andreea Dutulescu, Stefan Ruseti, Mihai Masala, Traian Rebedea, Mihai Dascalu
llmsnlpreinforcement-learningbenchmarks
More like this →