arXiv AI September 17, 2026 By Peter Chen, Xi Chen, Wotao Yin, Tianyi Lin

A Zeroth-Order Paradigm for LLM Preference Alignment

The paper introduces Comparison-based Preference Optimization (ComPO), a zeroth-order method that aligns large language models with human preferences using comparison oracles instead of direct differentiable loss optimization. It provides theoretical convergence guarantees for both offline and online variants under smoothness, gradient sparsity, and oracle compatibility assumptions, and establishes performance bounds under local coverage and in-distribution reward accuracy. Experiments on several LLMs (Mistral, Llama, Gemma-2, Qwen3, Gemma-3) show that ComPO outperforms existing direct alignment methods, achieving higher length-controlled win rates and diagnostics that suggest mitigation of likelihood displacement.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv Machine Learning

Sep 10

Towards Bridging the Gap Between Offline and Iterative Alignment via Preference Distillation

arXiv:2609.06893v1 Announce Type: cross Abstract: Direct preference optimization DPO is a promising offline approach for aligning large language models (LLMs) due to its simplicity, computational eff...

By Wenbo Zhang, Wenzhuo Zhou, Hengrui Cai, Zhengling Qi

llms reinforcement-learning efficiency benchmarks safety

arXiv AI

Aug 18

Leveraging Machine Unlearning for Cost-Efficient Preference Alignment

arXiv:2504. 06659v2 Announce Type: replace-cross Abstract: Despite advances in Preference Alignment (PA) for Large Language Models (LLMs), mainstream methods like reinforcement learning with human feedback face notable challenges.

By Xiaohua Feng, Yuyuan Li, Huwei Ji, Jiaming Zhang, Li Zhang, Tianyu Du, Chaochao Chen

llms reinforcement-learning safety

arXiv AI

Jun 11

TokenRatio: Principled Token-Level Preference Optimization via Ratio Matching

arXiv:2605. 12288v3 Announce Type: replace-cross Abstract: Direct Preference Optimization (DPO) is a widely used RL-free method for aligning language models from pairwise preferences, but it models preferences over full sequences even though generation is driven by per-token decisions.

By Truong Nguyen, Tien-Phat Nguyen, Linh Ngo Van, Duy Minh Ho Nguyen, Khoa Doan, Trung Le

llms nlp reinforcement-learning benchmarks safety

arXiv AI

Sep 10

DSPA: Dynamic SAE Steering for Data-Efficient Preference Alignment

The paper introduces DSPA, a dynamic sparse autoencoder (SAE) steering technique that aligns language model outputs with user preferences during inference, avoiding costly weight updates. DSPA constructs a conditional-difference map from preference triples to adjust token-active latents, improving MT‑Bench scores and matching AlpacaEval performance on models like Gemma‑2 and Qwen3 while preserving accuracy. It demonstrates robustness with limited preference data, outperforms the two‑stage RAHF‑SCIT pipeline in FLOPs, and reveals that preference directions are largely driven by discourse and stylistic cues.

By James Wedgwood, Aashiq Muhamed, Mona T. Diab, Virginia Smith

benchmarks safety

arXiv Machine Learning

Jul 7

Shapley-based Data Valuation for LLM Alignment via Sequential Preference Optimization

arXiv:2512. 15765v3 Announce Type: replace Abstract: Data valuation is a natural framework for understanding which preference datasets matter most when aligning a Large Language Model (LLM) using multiple sources.

By M\'elissa Tamine, Otmane Sakhi, Benjamin Heymann, Maxime Vono, Patrick Loiseau

llms reinforcement-learning fine-tuning safety

arXiv AI

Jun 11

Toward Preference-aligned Large Language Models via Residual-based Model Steering

arXiv:2509. 23982v2 Announce Type: replace-cross Abstract: Preference alignment is a critical step in making Large Language Models (LLMs) useful and aligned with (human) preferences.

By Lucio La Cava, Andrea Tagarelli

llms reinforcement-learning benchmarks safety

One email a morning, machine-written

Related stories

Towards Bridging the Gap Between Offline and Iterative Alignment via Preference Distillation

Leveraging Machine Unlearning for Cost-Efficient Preference Alignment

TokenRatio: Principled Token-Level Preference Optimization via Ratio Matching

DSPA: Dynamic SAE Steering for Data-Efficient Preference Alignment

Shapley-based Data Valuation for LLM Alignment via Sequential Preference Optimization

Toward Preference-aligned Large Language Models via Residual-based Model Steering