arXiv Machine Learning October 22, 2026 By Saman Jayasinghe, Sean Walton, Alma Rahat, Andy Gray, Chris Wheadon

Alternative Strategies for Active Learning with Bayesian Bradley Terry Model

Read the original on arXiv Machine Learning →

The paper introduces a Bayesian Bradley‑Terry model (BBTM) that employs Polya‑Gamma data augmentation for efficient Gibbs sampling and quantifies uncertainty in latent abilities and ranks. Three active‑learning strategies based on the knowledge‑gradient principle are proposed, using Kendall’s tau distance, posterior rank entropy, and posterior separation as acquisition functions, with approximate one‑step updates to reduce computational cost. The authors also present Posterior Separation Reliability (PSR) as an internal indicator of ranking reliability, and demonstrate through synthetic experiments that BBTM outperforms preference‑based Bayesian comparative judgement, with the entropy‑based strategy excelling at low‑to‑moderate noise and the posterior‑separation strategy at high noise levels.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Hugging Face Trending Papers

Oct 21

Alternative Strategies for Active Learning with Bayesian Bradley Terry Model

Comparative judgement uses pairwise comparisons to rank items, but recovering accurate rankings from a limited number of judgements depends on both the inference method and the pairs selected. We deve...

arXiv AI

Jul 29

Laplace-PSN-IRT: Uncertainty Quantification for Neural Item Response Theory Models of LLM Benchmarks

arXiv:2607. 25257v1 Announce Type: cross Abstract: Item Response Theory (IRT) has recently been proposed as a framework for evaluating large language model (LLM) benchmarks by separating a model's latent ability from the properties of individual benchmark items.

By Juan Francisco, Mandujano Reyes

llms benchmarks

arXiv AI

Sep 15

Confident Rankings with Fewer Items: Adaptive LLM Evaluation with Continuous Scores

arXiv:2601.13885v2 Announce Type: replace-cross Abstract: Computerized Adaptive Testing (CAT) has proven effective for efficient LLM evaluation on multiple-choice benchmarks, but modern LLM evaluatio...

By Esma Balk{\i}r, Alice Pernthaller, Marco Basaldella, Jos\'e Hern\'andez-Orallo, Nigel Collier

llms rag benchmarks

arXiv Machine Learning

Jul 30

BayesAME: Bayesian Active Model Evaluation

arXiv:2607. 27023v1 Announce Type: new Abstract: Evaluating large generative models across benchmarks is time-consuming and computationally expensive.

By Paula Cordero Encinar, Taylan Cemgil, Arnaud Doucet, Virginia Aglietti, Silvia Chiappa

benchmarks

arXiv Machine Learning

Jul 3

Ask the Right Comparison:Bias-Aware Bayesian Active Top-$k$ Ranking with LLM Judges

arXiv:2607. 02104v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used as cheap, scalable judges that compare candidate outputs pairwise -- to rank responses, select models, or triage papers.

By Jian Xu, Delu Zeng, John Paisley, Qibin Zhao

llms benchmarks safety

arXiv Statistics ML

Sep 4

Low Rank for Rank: Uncertainty-Aware Task-Specific LLM Ranking under Sparse Pairwise Comparisons

The paper introduces a low‑rank framework for ranking large language models (LLMs) on task‑specific benchmarks using sparse pairwise comparisons. By modeling the task‑by‑model ability matrix as low rank, the method shares information across related tasks while preserving task‑specific differences, and it provides uncertainty‑aware ranking through debiased estimators and simultaneous confidence sets. Experiments on synthetic data and the Chatbot Arena benchmark demonstrate improved sample efficiency and tighter, better‑calibrated ranking certificates, especially in the sparse comparison regime typical of real LLM evaluations.

By Jiachun Li, David Simchi-Levi, Will Wei Sun

llms benchmarks

One email a morning, machine-written

Related stories

Alternative Strategies for Active Learning with Bayesian Bradley Terry Model

Laplace-PSN-IRT: Uncertainty Quantification for Neural Item Response Theory Models of LLM Benchmarks

Confident Rankings with Fewer Items: Adaptive LLM Evaluation with Continuous Scores

BayesAME: Bayesian Active Model Evaluation

Ask the Right Comparison:Bias-Aware Bayesian Active Top-$k$ Ranking with LLM Judges

Low Rank for Rank: Uncertainty-Aware Task-Specific LLM Ranking under Sparse Pairwise Comparisons