← Back to all news
arXiv Machine Learning By Kazusato Oko, Annie Ulichney, Nika Haghtalab, Han Bao

Distortion of AI Alignment Revisited: RLHF is a Decent Utilitarian Aligner

Read the original on arXiv Machine Learning →

The paper revisits the distortion problem in Reinforcement Learning from Human Feedback (RLHF) for aligning large language models. It shows that exponential degradation in user utility, previously attributed to RLHF, actually stems from a mismatch between the preference data distribution and the KL reference policy. By deriving tight bounds across different KL regularization regimes, the authors demonstrate that when the two distributions match, RLHF achieves near‑optimal distortion, and they recommend using on‑policy data or pre‑fine‑tuning on data close to the true preference distribution.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Aug 4

Tail-Aware Information-Theoretic Bounds for LLM Alignment under Heavy-Tailed Rewards

arXiv:2604. 10727v2 Announce Type: replace-cross Abstract: Classical information-theoretic learning bounds typically rely on KL mutual information and moment-generating-function (MGF) arguments, which are well matched to bounded or sub-Gaussian losses but can be ineffective when losses or rewards are heavy-tailed.

By Huiming Zhang, Binghan Li, Wan Tian, Qiang Sun
llmsreinforcement-learningsafety
More like this →
arXiv AI
Jun 30

Pessimism's Paradox: Conservative Offline Training Amplifies Reward Hacking During Online Adaptation in Reasoning Models

arXiv:2606. 30627v1 Announce Type: cross Abstract: Conservative offline training is widely advocated as a safe foundation for subsequent online adaptation: if a policy stays close to well-supported behaviour, the argument goes, it is less likely to exploit imperfections in a learned reward model.

By Subramanyam Sahoo, Aman Chadha, Vinija Jain, Divya Chaudhary
reinforcement-learningsafety
More like this →
arXiv Machine Learning
Aug 28

Privacy Without Regret: Differentially Private Inference-Time Alignment

The paper introduces Private Best-of-N (PrivBoN), a method that adds calibrated Gumbel noise to reward scores during inference-time alignment, achieving both ε-differential privacy and KL-regularized alignment. When the privacy budget exceeds a critical threshold ε*, the noise becomes regret-optimal, matching the theoretical alignment skyline. The authors also propose Private Inference-Time Pessimism (PrivITP), which uses χ^2-regularized rejection sampling and a two-phase Gaussian mechanism to provide ex-post (ε,δ)-DP with a privacy cost independent of the number of responses, and demonstrate that both methods outperform standard Best-of-N across multiple models and datasets.

By Ishi Jain, Nandini Bhattad, Sayak Ray Chowdhury
llmsreinforcement-learningsafety
More like this →