arXiv Machine Learning

Adaptive Doubly Robust Off-Policy Evaluation for Ranking Policies under Diverse User Behavior

The paper introduces Adaptive Doubly Robust (ADR), an off‑policy evaluation method for ranking policies that blends adaptive importance weighting with reward regression to reduce variance. ADR is unbiased when the true user behavior model is known and, under a sufficient condition, achieves lower variance than the prior Adaptive Inverse Propensity Scoring (AIPS) approach. Experiments on synthetic data show that ADR consistently improves mean squared error over AIPS and other ranking OPE estimators across various data sizes and ranking lengths.

arXiv Machine Learning
6d ago

Prediction-Powered LLM Ranking with Adaptively Collected Human Feedback

The paper introduces Prediction-Powered Ranking (PP‑Rank), a framework that blends large language model (LLM) judge predictions with limited human feedback to rank LLMs more precisely while maintaining valid inference under adaptive sampling. PP‑Rank uses data‑driven prediction weights to minimize asymptotic covariance, establishes asymptotic normality for general evaluation targets, and provides a rule for reducing exploration that balances cost and variance. Experiments on UltraFeedback‑Extended and synthetic data show that PP‑Rank improves score estimation, narrows confidence intervals, certifies rankings, and can cut evaluation costs without losing precision.

By Gabriel Sargent, Will Wei Sun, Zhengwu Zhang, Yufeng Liu
arXiv Machine Learning
Oct 8

Efficient Best-of-N policy evaluation for inference-time alignment

The paper introduces a sample‑only framework for evaluating and selecting Best‑of‑N (BoN) inference‑time alignment policies without needing response likelihoods. By exploiting the order‑statistic structure of BoN, the authors express required density ratios through score‑rank probabilities that can be estimated from samples alone, and develop a doubly robust estimator (BoN‑DR) that reuses a shared auxiliary sample pool across candidate budgets. The method provides valid asymptotic inference even with reward estimator misspecification, proves efficiency, and offers two selection rules—maximizing estimated policy value or a lower confidence bound—to avoid over‑optimization and guarantee no harm, as demonstrated on synthetic data and GSM8K.

By Jonas Schweisthal, Yuxin Wang, Athiya Deviyani, Stefan Feuerriegel, Dennis Frauen
arXiv Statistics ML
Nov 9

High-Dimensional Off-Policy Evaluation without Sample Splitting using an Onsager Correction

The paper introduces an Onsager‑corrected, target‑policy‑weighted criterion that estimates the out‑of‑sample risk of a reward model trained on the full dataset using only its training data, thereby eliminating the need for sample splitting. This approach allows early stopping along a full‑data gradient‑descent trajectory and provides uniform consistency guarantees for neural networks under Gaussian contexts and single‑index rewards. The correction also yields a doubly robust policy‑value estimator that remains consistent even when reward‑model prediction error does not vanish, and experiments demonstrate reduced risk‑estimation error and improved reward‑model selection compared to held‑out baselines. whyItMatters":"By removing the need for sample splitting, the method enables more efficient use of data and improves the reliability of off‑policy evaluation and reward‑model selection in high‑dimensional settings."

By Saksham Jain, Guillaume Braun, Masaaki Imaizumi
arXiv AI
3d ago

When Both Sides Adapt: The Challenges of Evaluating Interactive LLM Systems

The paper investigates how user adaptivity in interactive LLM systems affects evaluation outcomes. It distinguishes between the common‑distribution effect (model performance on a fixed prompt set) and the induced‑distribution effect (performance on prompts generated by user interaction). The authors show that offline benchmarks and live A/B tests target different estimands, can rank models differently, and that user adaptivity can increase estimation variance and lead to challenges such as reward hacking and novelty effects.

By Hannah Li