← Back to all news
arXiv Machine Learning By Younwoo Choi, Leo Feng, Vincent Liu, Haanvid Lee

Amortized Off-Policy Evaluation for LLMs

Read the original on arXiv Machine Learning →

The paper introduces PFN‑OPE, a prior‑data fitted network that performs amortized off‑policy evaluation (OPE) for large language models (LLMs). Unlike traditional OPE methods that must be retrained for each new logged dataset or reward definition, PFN‑OPE is pretrained once on a diverse set of contextual‑bandit tasks and can estimate value for any new logged dataset and target response in a single forward pass. Experiments on HelpSteer2 and UltraFeedback with Qwen, Llama, and Gemma policies show that PFN‑OPE reduces error by 2.0 to 9.3 times compared to the best baselines in reward‑shifted settings.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Machine Learning
Sep 29

Offline Policy Evaluation as a decision support tool for designing Adaptive Experiments

The paper explores how data from fixed A/B tests can guide the deployment of adaptive experiments using contextual bandits. By combining off‑policy evaluation with a controlled warm‑start simulation, the authors rank pre‑specified adaptive and non‑adaptive policies using doubly robust estimators. Experiments on synthetic trials and real benchmarks show that adaptive, context‑aware policies outperform fixed allocations when heterogeneity exists, but offer little advantage otherwise.

By Jo\~ao Victor Ferreira Alves, Eduardo Rocha Laurentino, Gustavo de Oliveira Kanno, Thiago Costa Rizuti da Rocha
benchmarks
More like this →
arXiv Machine Learning
Aug 13

When Offline Evaluation Misleads: A Diagnostic Protocol for Reward and Policy Selection in Delayed-Feedback Contextual Bandits

arXiv:2608. 11560v1 Announce Type: new Abstract: Personalizing marketing messages with contextual multi-armed bandits (CMABs) drives real business value, yet the objective that ultimately matters - a downstream conversion - is observed only weeks later, too late to drive online learning.

By Sang Su Lee, Vineeth Loganathan, Shishir Dash, Vijay Raghavan
reinforcement-learningbenchmarkssafety
More like this →
arXiv Machine Learning
Oct 15

Lag: A Mean-Field Optimal Policy for Online Assignment with Reusable Resources

The paper introduces Lag, a mean‑field optimal index policy for online assignment problems with reusable resources, such as peer review, call routing, ridesharing, emergency dispatch, and crowdsourcing. By modeling the problem as a contextual restless multi‑armed bandit and deriving an occupancy‑measure linear program, the authors obtain Lag from the dual and prove it is asymptotically optimal in a scaling regime where server copies and arrivals grow together. Experiments on peer‑review and call‑center data show Lag achieves 83–99% of the LP upper bound and outperforms greedy assignment by up to 36% in synthetic settings.

By Keerthana Gurushankar
reinforcement-learning
More like this →
Hugging Face Trending Papers
Aug 12

When Offline Evaluation Misleads: A Diagnostic Protocol for Reward and Policy Selection in Delayed-Feedback Contextual Bandits

Personalizing marketing messages with contextual multi-armed bandits (CMABs) drives real business value, yet the objective that ultimately matters - a downstream conversion - is observed only weeks later, too late to drive online learning. Teams therefore train the bandit on a fast proxy reward, and separately must judge whether a contextual bandit is worth its complexity over sending one best message.

reinforcement-learningbenchmarkssafety
More like this →