arXiv:2609.36740v1 Announce Type: new
Abstract: Many recommender systems such as for e-commerce and news platforms aim to provide users with rankings they are likely to interact with. Off-Policy Lear...
By Ren Kishimoto, Koichi Tanaka, Haruka Kiyohara, Yusuke Narita, Yasuo Yamamoto, Nobuyuki Shimizu, Yuta Saito
arXiv:2610.08677v1 Announce Type: new
Abstract: Off-policy evaluation (OPE) for contextual bandit policies becomes challenging when action-level importance weighting incurs excessive variance. Doubly...
By Nicol\`o Felicioni, Michael Benigni, Maurizio Ferrari Dacrema, Paolo Cremonesi
The paper introduces Prediction-Powered Ranking (PP‑Rank), a framework that blends large language model (LLM) judge predictions with limited human feedback to rank LLMs more precisely while maintaining valid inference under adaptive sampling. PP‑Rank uses data‑driven prediction weights to minimize asymptotic covariance, establishes asymptotic normality for general evaluation targets, and provides a rule for reducing exploration that balances cost and variance. Experiments on UltraFeedback‑Extended and synthetic data show that PP‑Rank improves score estimation, narrows confidence intervals, certifies rankings, and can cut evaluation costs without losing precision.
By Gabriel Sargent, Will Wei Sun, Zhengwu Zhang, Yufeng Liu
The paper introduces a sample‑only framework for evaluating and selecting Best‑of‑N (BoN) inference‑time alignment policies without needing response likelihoods. By exploiting the order‑statistic structure of BoN, the authors express required density ratios through score‑rank probabilities that can be estimated from samples alone, and develop a doubly robust estimator (BoN‑DR) that reuses a shared auxiliary sample pool across candidate budgets. The method provides valid asymptotic inference even with reward estimator misspecification, proves efficiency, and offers two selection rules—maximizing estimated policy value or a lower confidence bound—to avoid over‑optimization and guarantee no harm, as demonstrated on synthetic data and GSM8K.
By Jonas Schweisthal, Yuxin Wang, Athiya Deviyani, Stefan Feuerriegel, Dennis Frauen
Off-policy evaluation (OPE) for contextual bandit policies becomes challenging when action-level importance weighting incurs excessive variance. Doubly robust (DR) estimation remains unbiased under co...
arXiv:2602. 14914v3 Announce Type: replace Abstract: Off-policy evaluation (OPE) is essential for assessing ranking and recommendation systems without costly online interventions.
By Olivier Jeunen, Shashank Gupta
arXiv:2506. 06989v3 Announce Type: replace-cross Abstract: Learning-to-rank (LTR) systems commonly depend on implicit feedback, such as user clicks, because it is easy to collect and can serve as a valuable signal of user preferences.
By Md Aminul Islam, Kathryn Vasilaky, Elena Zheleva
The paper introduces an Onsager‑corrected, target‑policy‑weighted criterion that estimates the out‑of‑sample risk of a reward model trained on the full dataset using only its training data, thereby eliminating the need for sample splitting. This approach allows early stopping along a full‑data gradient‑descent trajectory and provides uniform consistency guarantees for neural networks under Gaussian contexts and single‑index rewards. The correction also yields a doubly robust policy‑value estimator that remains consistent even when reward‑model prediction error does not vanish, and experiments demonstrate reduced risk‑estimation error and improved reward‑model selection compared to held‑out baselines.
whyItMatters":"By removing the need for sample splitting, the method enables more efficient use of data and improves the reliability of off‑policy evaluation and reward‑model selection in high‑dimensional settings."
By Saksham Jain, Guillaume Braun, Masaaki Imaizumi
The paper investigates how user adaptivity in interactive LLM systems affects evaluation outcomes. It distinguishes between the common‑distribution effect (model performance on a fixed prompt set) and the induced‑distribution effect (performance on prompts generated by user interaction). The authors show that offline benchmarks and live A/B tests target different estimands, can rank models differently, and that user adaptivity can increase estimation variance and lead to challenges such as reward hacking and novelty effects.
By Hannah Li
arXiv:2607. 16240v1 Announce Type: cross Abstract: Direct Alignment Algorithms (DAAs) such as DPO have become a common way to post-train and align LLMs with human preferences.
By Shawn Im, Federico Danieli, Skyler Seto, Barry-John Theobald, Katherine Metcalf
arXiv:2512.23067v3 Announce Type: replace
Abstract: Personalized alignment from preference data has focused primarily on improving personal reward model (RM) accuracy, with the implicit assumption th...
By Fady Rezk, Yuangang Pan, Chuan-Sheng Foo, Xun Xu, Nancy Chen, Henry Gouk, Timothy Hospedales
arXiv:2609.38860v1 Announce Type: cross
Abstract: Learning from human preferences is central to large language model (LLM) alignment, but human preference annotation is costly. Active preference lear...
By Zhongman Du, Huiming Zhang, Haodong Zhu, Baochang Zhang