The paper introduces MoPLEx, an expectation‑maximization algorithm for learning mixtures of Plackett‑Luce models from multi‑way ranking data. It augments rankings with synthetic responses from a base language model and uses a gradient‑based estimation to reduce inference cost, enabling efficient fitting of large‑scale models. Experiments show the method achieves low probability estimation error and improves clustering and ranking accuracy by 43.7% and 15.2% over baselines.
arXiv:2608.25200v2 Announce Type: replace-cross Abstract: We study learning a mixture of $k$ Plackett-Luce models from multi-way ranking responses from annotators that may represent heterogeneous und...
arXiv:2601. 21816v2 Announce Type: replace Abstract: Evaluating the performance of large language models (LLMs) from human preference data is crucial for obtaining LLM leaderboards.
arXiv:2504. 06659v2 Announce Type: replace-cross Abstract: Despite advances in Preference Alignment (PA) for Large Language Models (LLMs), mainstream methods like reinforcement learning with human feedback face notable challenges.
Aligning large language models to human preferences is crucial for real-world deployment but frequently incurs an alignment tax, leading to the catastrophic forgetting of pre-trained general capabilit...
arXiv:2602. 02898v3 Announce Type: replace Abstract: Language model benchmarks are pervasive and computationally-efficient proxies for real-world performance.
Aligning large language models with diverse and heterogeneous human values requires multi-objective alignment methods to effectively trade off conflicting preference dimensions. Current methods achieve this trade-off by training policies conditioned on preference vectors and leveraging online direct preference optimization.
The paper introduces BALIGN, a balanced data selection strategy designed to reduce catastrophic forgetting—referred to as the alignment tax—in large language models during preference-based alignment. By analyzing preference optimization gradients, the authors identify three data-centric features that influence parameter drift: the reference model's log-probability margin, token length differences between chosen and rejected responses, and TF‑IDF similarity to general capability corpora. BALIGN aggregates these features into a composite risk score to filter out high-risk preference samples, thereby preserving foundational capabilities while maintaining alignment gains with minimal computational overhead.
The paper introduces Comparison-based Preference Optimization (ComPO), a zeroth-order method that aligns large language models with human preferences using comparison oracles instead of direct differentiable loss optimization. It provides theoretical convergence guarantees for both offline and online variants under smoothness, gradient sparsity, and oracle compatibility assumptions, and establishes performance bounds under local coverage and in-distribution reward accuracy. Experiments on several LLMs (Mistral, Llama, Gemma-2, Qwen3, Gemma-3) show that ComPO outperforms existing direct alignment methods, achieving higher length-controlled win rates and diagnostics that suggest mitigation of likelihood displacement.
arXiv:2608.23149v1 Announce Type: cross Abstract: The alignment of Large Language Models heavily relies on English-centric high-quality preference data, which often leads to suboptimal performance in...
UniPolicy is a multi-policy alignment framework for search advertising that jointly optimizes relevance, click propensity, and commercial value. It uses objective-specific prefix tokens, sparse MoE-LoRA routing, and residual FFNs to decouple parameters within a shared backbone, and builds pairwise preferences from multi-stage behavioral feedback to improve generation. In large-scale offline tests and a 7‑day online A/B test, UniPolicy achieved balanced gains across metrics, boosting CTR by 0.71%, RPS by 1.58%, and revenue by 1.32% while keeping latency stable.
The paper introduces GAP-DPO, a method for personalizing large language models by selecting preference pairs based on gradient alignment with user utility. It formalizes personalized preference learning as a geometry‑aligned optimization problem, showing that off‑policy sampling can shift DPO updates from error correction to reinforcement when preference margins align with utility gradients. Experiments demonstrate that GAP‑DPO improves stylistic fidelity, preference alignment, and overall generation quality over standard DPO variants.
arXiv:2510. 05342v2 Announce Type: replace-cross Abstract: Direct Preference Optimization (DPO) has emerged as a simple and effective method for aligning large language models.