arXiv Statistics ML

Low Rank for Rank: Uncertainty-Aware Task-Specific LLM Ranking under Sparse Pairwise Comparisons

The paper introduces a low‑rank framework for ranking large language models (LLMs) on task‑specific benchmarks using sparse pairwise comparisons. By modeling the task‑by‑model ability matrix as low rank, the method shares information across related tasks while preserving task‑specific differences, and it provides uncertainty‑aware ranking through debiased estimators and simultaneous confidence sets. Experiments on synthetic data and the Chatbot Arena benchmark demonstrate improved sample efficiency and tighter, better‑calibrated ranking certificates, especially in the sparse comparison regime typical of real LLM evaluations.

arXiv AI

1d ago

LLM Evaluation as Tensor Completion: Low Rank Structure and Semiparametric Efficiency

The paper treats large language model (LLM) evaluation as a tensor completion problem, modeling noisy, sparse, and non‑uniform pairwise human judgments through a low‑rank latent score tensor under Bradley‑Terry‑Luce‑type models. It derives the efficient influence function and semiparametric efficiency bound for smooth functionals of the true tensor, and proposes a one‑step debiased estimator with asymptotic normality. A key innovation is a score‑whitening technique that equalizes local Fisher information, overcoming anisotropy in the information operator and enabling stable inference at optimal sample‑complexity.

By Jiachun Li, David Simchi-Levi, Will Wei Sun

llms

arXiv Machine Learning

Jun 9

Rank Intervals for Leaderboards: A Hierarchical Framework for Model Evaluation

arXiv:2606. 08679v1 Announce Type: cross Abstract: Pretrained models are often evaluated on multi-task leaderboards to measure their applicability in diverse contexts.

By Bitya Neuhof, Yuval Benjamini

benchmarks

arXiv Machine Learning

Jul 23

Statistical Inference for Rank Allocation in Low-Rank Adaptation

arXiv:2607. 20205v1 Announce Type: cross Abstract: Low-rank adaptation (LoRA) has become a widely used parameter-efficient fine-tuning method for large language models.

By Yihang Gao, Vincent Y. F. Tan

llms nlp fine-tuning

Hugging Face Trending Papers

Jul 22

Statistical Inference for Rank Allocation in Low-Rank Adaptation

Low-rank adaptation (LoRA) has become a widely used parameter-efficient fine-tuning method for large language models. Since different modules and layers may contribute unequally to downstream adaptation, allocating rank resources under a fixed parameter budget is an important problem for balancing efficiency, expressiveness, and generalization.

llms nlp fine-tuning

arXiv Machine Learning

Jun 30

BaRA: Bayesian Adaptive Rank Allocation for Parameter-Efficient Fine-Tuning

arXiv:2606. 29184v1 Announce Type: new Abstract: While Low-rank adaptation (LoRA) enables highly efficient fine-tuning by constraining task-specific updates to fixed low-rank subspaces, this rigid design limits representational flexibility and often results in overconfident predictions and miscalibrated uncertainty, especially in low-data regimes.

By Zhibin Duan, Yuhong Wang, Jiahong Fu, Zongsheng Yue, Bo Chen, Zongben Xu

fine-tuning benchmarks

arXiv Machine Learning

Jun 5

A Judge-Aware Ranking Framework for Evaluating Large Language Models without Ground Truth

arXiv:2601. 21817v2 Announce Type: replace-cross Abstract: Evaluating large language models (LLMs) on open-ended tasks without ground-truth labels is increasingly done via the LLM-as-a-judge paradigm.

By Mingyuan Xu, Xinzi Tan, Jiawei Wu, Doudou Zhou

llms benchmarks

arXiv AI

Jul 29

Laplace-PSN-IRT: Uncertainty Quantification for Neural Item Response Theory Models of LLM Benchmarks

arXiv:2607. 25257v1 Announce Type: cross Abstract: Item Response Theory (IRT) has recently been proposed as a framework for evaluating large language model (LLM) benchmarks by separating a model's latent ability from the properties of individual benchmark items.

By Juan Francisco, Mandujano Reyes

llms benchmarks

arXiv Machine Learning

Jul 3

Bayesian Sparse Low-Rank Adaptation for Large Language Model Uncertainty Estimation

arXiv:2607. 02182v1 Announce Type: new Abstract: Large language models (LLMs) exhibit remarkable reasoning capabilities, but their task-specific fine-tuning is notoriously plagued by overconfidence, severely hindering trustworthy deployment.

By Jijie Zhang, Zhe Ren, Quan Zhang, Dandan Guo

llms fine-tuning

arXiv AI

4d ago

Balance of Benchmarks: Semantic Density Reweighting for Benchmark Multiplicity and Task-Conditioned Evaluation

arXiv:2608.30044v1 Announce Type: new Abstract: Language models are commonly compared by averaging scores across a benchmark list with equal weight. Such lists grow through publication outside an exp...

By Jhen-Ke Lin

llms benchmarks

arXiv Machine Learning

Jun 29

Benchmarking on Tasks That Matter: Dataset Selection for Preserving Model Rankings

arXiv:2606. 27997v1 Announce Type: new Abstract: Benchmarks of machine learning models often include many datasets, making evaluation expensive.

By Rostislav Gusev, Alexey Zaytsev

nlp benchmarks

arXiv AI

Aug 25

Rank Reversal in Multilingual LLM Judges: A Label-Free Double-Centering Calibrator

arXiv:2608.22432v1 Announce Type: cross Abstract: Multilingual LLM judges produce different evaluator-backbone rankings depending on the prompt language: on an eight-language Agent-as-a-Judge benchma...

By Alhasan Mahmood, Samir Abdaljalil, Hasan Kurban

llms agents benchmarks

arXiv Machine Learning

Aug 4

Aggregate-then-Calibrate for Human-centered Assessment with Theoretical Guarantees

arXiv:2608. 02455v1 Announce Type: new Abstract: Human-centered assessment tasks, which are essential for systematic decision-making, rely heavily on human judgment and typically lack verifiable ground truth.

By Zejun Xie, Xintong Li, Guang Wang, Desheng Zhang

Related stories