← Back to all news
arXiv Machine Learning By Debangshu Banerjee, Kintan Saha, Aditya Gopalan

Risk-Aware Alignment Must Incorporate Reward Uncertainty

Read the original on arXiv Machine Learning →

The paper addresses policy optimization when rewards are noisy, restricting the policy to stay near a reference policy. It defines risk as the probability of ending up with a worse policy than the reference and shows that, under a Gaussian reward model, the optimal regularizer is a quadratic norm weighted by the reward covariance. Using this insight, the authors propose a new algorithm that regularizes with reward variance estimates, and experiments on synthetic bandits, large language models, and reward models demonstrate that it yields more robust alignment outcomes than existing methods.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
Oct 19

GVPO: Group Variance Policy Optimization for Large Language Model Post-Training

GVPO: Group Variance Policy Optimization is a new post‑training method for large language models that integrates the analytical solution to KL‑constrained reward maximization into its gradient weights. It guarantees a unique optimal solution and allows flexible sampling distributions, avoiding the on‑policy and importance sampling limitations of previous techniques. The approach offers a theoretical guarantee while remaining practically adaptable, positioning it as a reliable and versatile paradigm for LLM post‑training.

By Kaichen Zhang, Yuzhong Hong, Junwei Bao, Hongfei Jiang, Yang Song, Dingqian Hong, Hui Xiong
llmssafety
More like this →
arXiv AI
Oct 28

Doubly Robust Alignment for Large Language Models

The paper introduces a doubly robust preference optimization algorithm for reinforcement learning from human feedback (RLHF) in large language models. It addresses sensitivity to misspecifications in the preference model, reference policy, or reward function by ensuring consistency when either the preference model or the reference policy is correctly specified. Experiments show that the method outperforms state‑of‑the‑art algorithms both theoretically and empirically.

By Erhan Xu, Kai Ye, Hongyi Zhou, Luhan Zhu, Francesco Quinzan, Chengchun Shi
llmsreinforcement-learningfine-tuningbenchmarkssafety
More like this →