arXiv Statistics ML

High-Dimensional Off-Policy Evaluation without Sample Splitting using an Onsager Correction

The paper introduces an Onsager‑corrected, target‑policy‑weighted criterion that estimates the out‑of‑sample risk of a reward model trained on the full dataset using only its training data, thereby eliminating the need for sample splitting. This approach allows early stopping along a full‑data gradient‑descent trajectory and provides uniform consistency guarantees for neural networks under Gaussian contexts and single‑index rewards. The correction also yields a doubly robust policy‑value estimator that remains consistent even when reward‑model prediction error does not vanish, and experiments demonstrate reduced risk‑estimation error and improved reward‑model selection compared to held‑out baselines. whyItMatters":"By removing the need for sample splitting, the method enables more efficient use of data and improves the reliability of off‑policy evaluation and reward‑model selection in high‑dimensional settings."

arXiv AI
Oct 1

Revisiting scaling laws for reward optimization

The paper presents a new scaling law for reward optimization in AI alignment, showing that performance scales as Θ(√min{log(M), K}), where M is the number of preference comparisons used to train a proxy reward model and K is the KL‑divergence budget relative to a reference policy. The authors derive this law using an information‑theoretic model, prove its tightness, and validate it with extensive experiments involving a 70B gold reward model and smaller proxy models (0.6B–4B). The empirical results demonstrate a strong fit (R² 97–99 %) across different model sizes, noise levels, and optimization methods, suggesting that reward optimization behaves like a simple selection task over IID Gaussian variables with noisy feedback.

By Ali Aouad, Aymane El Gadarri, Vivek F. Farias
arXiv Machine Learning
Oct 9

Amortized Off-Policy Evaluation for LLMs

The paper introduces PFN‑OPE, a prior‑data fitted network that performs amortized off‑policy evaluation (OPE) for large language models (LLMs). Unlike traditional OPE methods that must be retrained for each new logged dataset or reward definition, PFN‑OPE is pretrained once on a diverse set of contextual‑bandit tasks and can estimate value for any new logged dataset and target response in a single forward pass. Experiments on HelpSteer2 and UltraFeedback with Qwen, Llama, and Gemma policies show that PFN‑OPE reduces error by 2.0 to 9.3 times compared to the best baselines in reward‑shifted settings.

By Younwoo Choi, Leo Feng, Vincent Liu, Haanvid Lee
arXiv Machine Learning
6d ago

High Probability Risk Control for Online Policy Learning

The paper introduces a method for controlling risk in online policy learning, ensuring that an agent’s policy risk remains below a specified threshold at every round with high probability. It tackles the challenges posed by feedback covariate shift by proposing a permutation‑based weighted risk evaluator, efficiently approximated via the Sinkhorn algorithm, and a Gaussian bootstrap‑derived upper confidence bound for policy selection. Theoretical results guarantee low‑variance risk estimation and asymptotic high‑probability risk control, while empirical tests on contextual bandits, safe question answering, and medical claim filtering demonstrate maintained risk levels, lower risk than baselines, and reliable policy selection.

By Yihong Guo, Drew Prinster, Suchi Saria, Anqi Liu
arXiv Machine Learning
Oct 2

Robust Risk-Sensitive Reinforcement Learning from Corrupted Human Feedback

The paper introduces WSP-CVaR-RLHF, a risk‑sensitive reinforcement learning algorithm that learns from human feedback while handling corrupted or adversarial preference labels. It uses uncertainty‑weighted reward estimation and optimistic CVaR planning to achieve a regret bound that separates clean statistical cost from corruption penalties. Experiments with four adversarial attacks show that the method consistently reduces cumulative regret compared to an unweighted robust baseline while maintaining confidence‑set coverage.

By Xinyi Ni, Lifeng Lai
arXiv Machine Learning
Jun 2

Bandit Simulation for Average Reward Inference

arXiv:2606. 00913v1 Announce Type: cross Abstract: Multi-arm bandit algorithms are increasingly used in online platforms, clinical trials, and social science experiments, but valid statistical inference on their performance remains an open challenge.

By Samya Praharaj, Chih-Yu Chang, Koulik Khamaru, Kelly W. Zhang
arXiv Machine Learning
Sep 29

Offline Policy Evaluation as a decision support tool for designing Adaptive Experiments

The paper explores how data from fixed A/B tests can guide the deployment of adaptive experiments using contextual bandits. By combining off‑policy evaluation with a controlled warm‑start simulation, the authors rank pre‑specified adaptive and non‑adaptive policies using doubly robust estimators. Experiments on synthetic trials and real benchmarks show that adaptive, context‑aware policies outperform fixed allocations when heterogeneity exists, but offer little advantage otherwise.

By Jo\~ao Victor Ferreira Alves, Eduardo Rocha Laurentino, Gustavo de Oliveira Kanno, Thiago Costa Rizuti da Rocha
arXiv AI
Nov 5

GIST: Gibbs-Targeted Hierarchical Data Selection for Offline RLVR

The paper introduces GIST, a Gibbs‑Targeted Hierarchical Data Selection method for offline reinforcement learning with verifiable rewards (RLVR). By analyzing the bidirectional KL divergence between a policy and its Gibbs‑optimal target, GIST selectively supervises correct responses on‑policy and incorrect responses near the policy’s mode, while focusing token supervision on mismatched tokens. Experiments on GSM8K, MATH‑500, and MBPP show that GIST improves average accuracy across three models and outperforms seven baselines on Qwen3‑0.6B/GSM8K, all while supervising 66.6% fewer tokens.

By Xuliang Huang, Tianxiang Zhao, Chengwei Qin, Yao Shu