The paper presents a new scaling law for reward optimization in AI alignment, showing that performance scales as Θ(√min{log(M), K}), where M is the number of preference comparisons used to train a proxy reward model and K is the KL‑divergence budget relative to a reference policy. The authors derive this law using an information‑theoretic model, prove its tightness, and validate it with extensive experiments involving a 70B gold reward model and smaller proxy models (0.6B–4B). The empirical results demonstrate a strong fit (R² 97–99 %) across different model sizes, noise levels, and optimization methods, suggesting that reward optimization behaves like a simple selection task over IID Gaussian variables with noisy feedback.
By Ali Aouad, Aymane El Gadarri, Vivek F. Farias
The paper introduces PFN‑OPE, a prior‑data fitted network that performs amortized off‑policy evaluation (OPE) for large language models (LLMs). Unlike traditional OPE methods that must be retrained for each new logged dataset or reward definition, PFN‑OPE is pretrained once on a diverse set of contextual‑bandit tasks and can estimate value for any new logged dataset and target response in a single forward pass. Experiments on HelpSteer2 and UltraFeedback with Qwen, Llama, and Gemma policies show that PFN‑OPE reduces error by 2.0 to 9.3 times compared to the best baselines in reward‑shifted settings.
By Younwoo Choi, Leo Feng, Vincent Liu, Haanvid Lee
arXiv:2609.37209v1 Announce Type: new
Abstract: Pairwise losses are increasingly used for reward learning even when pointwise rewards are observed, with mixed empirical results. When and why do pairw...
By Junghyun Lee, Minsoo Ha, Sanghwa Kim, Yeongjong Kim, Eunjee Lee, Seiyun Shin, Kwang-Sung Jun
arXiv:2609.37119v1 Announce Type: cross
Abstract: Recent approaches to reinforcement learning (RL) post-training for large language models increasingly remove the critic to reduce training instabilit...
By Hongyang Li, Xiao Li, Caesar Wu, Said Mammar, Gr\'egoire Danoy, Pascal Bouvry
The paper introduces a method for controlling risk in online policy learning, ensuring that an agent’s policy risk remains below a specified threshold at every round with high probability. It tackles the challenges posed by feedback covariate shift by proposing a permutation‑based weighted risk evaluator, efficiently approximated via the Sinkhorn algorithm, and a Gaussian bootstrap‑derived upper confidence bound for policy selection. Theoretical results guarantee low‑variance risk estimation and asymptotic high‑probability risk control, while empirical tests on contextual bandits, safe question answering, and medical claim filtering demonstrate maintained risk levels, lower risk than baselines, and reliable policy selection.
By Yihong Guo, Drew Prinster, Suchi Saria, Anqi Liu
arXiv:2603. 06957v2 Announce Type: replace-cross Abstract: We study post-training linear autoregressive models with outcome and process rewards.
By Alireza Mousavi-Hosseini, Murat A. Erdogdu
arXiv:2611.05942v1 Announce Type: new
Abstract: RL for LLM reasoning often reaches an early performance plateau as the policy distribution becomes increasingly sharp. Recent work has studied policy e...
By Mengni Jia, Mengyu Zhou, Xiaoxi Jiang, Guanjun Jiang
arXiv:2606. 01081v1 Announce Type: new Abstract: Decision-focused learning (DFL) trains predictive models by optimizing downstream decision quality rather than standalone prediction accuracy.
By Wyame Benslimane, Tinghan Ye, Pascal Van Hentenryck, Paul Grigas
The paper introduces WSP-CVaR-RLHF, a risk‑sensitive reinforcement learning algorithm that learns from human feedback while handling corrupted or adversarial preference labels. It uses uncertainty‑weighted reward estimation and optimistic CVaR planning to achieve a regret bound that separates clean statistical cost from corruption penalties. Experiments with four adversarial attacks show that the method consistently reduces cumulative regret compared to an unweighted robust baseline while maintaining confidence‑set coverage.
By Xinyi Ni, Lifeng Lai
arXiv:2606. 00913v1 Announce Type: cross Abstract: Multi-arm bandit algorithms are increasingly used in online platforms, clinical trials, and social science experiments, but valid statistical inference on their performance remains an open challenge.
By Samya Praharaj, Chih-Yu Chang, Koulik Khamaru, Kelly W. Zhang
The paper explores how data from fixed A/B tests can guide the deployment of adaptive experiments using contextual bandits. By combining off‑policy evaluation with a controlled warm‑start simulation, the authors rank pre‑specified adaptive and non‑adaptive policies using doubly robust estimators. Experiments on synthetic trials and real benchmarks show that adaptive, context‑aware policies outperform fixed allocations when heterogeneity exists, but offer little advantage otherwise.
By Jo\~ao Victor Ferreira Alves, Eduardo Rocha Laurentino, Gustavo de Oliveira Kanno, Thiago Costa Rizuti da Rocha
The paper introduces GIST, a Gibbs‑Targeted Hierarchical Data Selection method for offline reinforcement learning with verifiable rewards (RLVR). By analyzing the bidirectional KL divergence between a policy and its Gibbs‑optimal target, GIST selectively supervises correct responses on‑policy and incorrect responses near the policy’s mode, while focusing token supervision on mismatched tokens. Experiments on GSM8K, MATH‑500, and MBPP show that GIST improves average accuracy across three models and outperforms seven baselines on Qwen3‑0.6B/GSM8K, all while supervising 66.6% fewer tokens.
By Xuliang Huang, Tianxiang Zhao, Chengwei Qin, Yao Shu