arXiv Machine Learning By Yihong Guo, Drew Prinster, Suchi Saria, Anqi Liu

High Probability Risk Control for Online Policy Learning

Read the original on arXiv Machine Learning →

The paper introduces a method for controlling risk in online policy learning, ensuring that an agent’s policy risk remains below a specified threshold at every round with high probability. It tackles the challenges posed by feedback covariate shift by proposing a permutation‑based weighted risk evaluator, efficiently approximated via the Sinkhorn algorithm, and a Gaussian bootstrap‑derived upper confidence bound for policy selection. Theoretical results guarantee low‑variance risk estimation and asymptotic high‑probability risk control, while empirical tests on contextual bandits, safe question answering, and medical claim filtering demonstrate maintained risk levels, lower risk than baselines, and reliable policy selection.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Machine Learning
Oct 2

Robust Risk-Sensitive Reinforcement Learning from Corrupted Human Feedback

The paper introduces WSP-CVaR-RLHF, a risk‑sensitive reinforcement learning algorithm that learns from human feedback while handling corrupted or adversarial preference labels. It uses uncertainty‑weighted reward estimation and optimistic CVaR planning to achieve a regret bound that separates clean statistical cost from corruption penalties. Experiments with four adversarial attacks show that the method consistently reduces cumulative regret compared to an unweighted robust baseline while maintaining confidence‑set coverage.

By Xinyi Ni, Lifeng Lai
arXiv Statistics ML
Nov 9

High-Dimensional Off-Policy Evaluation without Sample Splitting using an Onsager Correction

The paper introduces an Onsager‑corrected, target‑policy‑weighted criterion that estimates the out‑of‑sample risk of a reward model trained on the full dataset using only its training data, thereby eliminating the need for sample splitting. This approach allows early stopping along a full‑data gradient‑descent trajectory and provides uniform consistency guarantees for neural networks under Gaussian contexts and single‑index rewards. The correction also yields a doubly robust policy‑value estimator that remains consistent even when reward‑model prediction error does not vanish, and experiments demonstrate reduced risk‑estimation error and improved reward‑model selection compared to held‑out baselines. whyItMatters":"By removing the need for sample splitting, the method enables more efficient use of data and improves the reliability of off‑policy evaluation and reward‑model selection in high‑dimensional settings."

By Saksham Jain, Guillaume Braun, Masaaki Imaizumi