Prediction-Powered LLM Ranking with Adaptively Collected Human Feedback
Read the original on arXiv Machine Learning →The paper introduces Prediction-Powered Ranking (PP‑Rank), a framework that blends large language model (LLM) judge predictions with limited human feedback to rank LLMs more precisely while maintaining valid inference under adaptive sampling. PP‑Rank uses data‑driven prediction weights to minimize asymptotic covariance, establishes asymptotic normality for general evaluation targets, and provides a rule for reducing exploration that balances cost and variance. Experiments on UltraFeedback‑Extended and synthetic data show that PP‑Rank improves score estimation, narrows confidence intervals, certifies rankings, and can cut evaluation costs without losing precision.
Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.