arXiv Machine Learning

Benchmarking on Tasks That Matter: Dataset Selection for Preserving Model Rankings

arXiv:2606. 27997v1 Announce Type: new Abstract: Benchmarks of machine learning models often include many datasets, making evaluation expensive.

arXiv Machine Learning

Jul 22

How Benchmark Prediction from Fewer Data Misses the Mark

arXiv:2506. 07673v2 Announce Type: replace Abstract: Large language model (LLM) evaluation is increasingly costly, prompting interest in methods that speed up evaluation by shrinking benchmark datasets.

By Guanhua Zhang, Florian E. Dorner, Moritz Hardt

llms benchmarks

arXiv AI

Jul 14

Coresets Before Score Sets: Evaluation-Unsupervised Prompt Subset Selection for LLM Benchmarks

arXiv:2607. 09739v1 Announce Type: new Abstract: We study LLM benchmark coreset selection: selecting a small subset of prompts over multiple benchmarks whose induced model scores and rankings approximate those obtained from the full benchmark suite.

By Jihan Yao, Gantavya Bhatt, Arnav Das, Peter Jin, Ke Bao, Qiaolin Yu, Khushi Bhardwaj, Chang Su, Jialei Wang, Yikai Zhu, Sugam Devare, Damon Mosk-Aoyama, Zhen Dong, Venkat Krishna Srinivasan, Yineng Zhang, Oleksii Kuchaiev, Jiantao Jiao, Banghua Zhu, Jeff Bilmes

llms rag benchmarks

arXiv Machine Learning

Jul 9

MTEB-BR: A Text Embedding Benchmark for Brazilian Portuguese

arXiv:2607. 04581v2 Announce Type: replace-cross Abstract: Text embeddings for Portuguese have no dedicated benchmark: evaluation rests on translated corpora such as English MS MARCO or on thin multilingual coverage, with native tasks scattered and unconsolidated.

By Tardelli Ronan Coelho Stekel

rag benchmarks

arXiv Machine Learning

Jul 7

MTEB-PT: A Text Embedding Benchmark for Brazilian Portuguese

arXiv:2607. 04581v1 Announce Type: cross Abstract: Text embeddings for Portuguese have no dedicated benchmark: evaluation rests on translated corpora such as English MS MARCO or on thin multilingual coverage, with native tasks scattered and unconsolidated.

By Tardelli Ronan Coelho Stekel

rag benchmarks

arXiv Machine Learning

Jun 8

Bradley-Terry Rankings for Recommender Systems Across Dataset Taxonomies

arXiv:2606. 07492v1 Announce Type: cross Abstract: The ranking of recommendation algorithms is a challenging problem since model performance is sensitive to dataset characteristics such as sparsity, sequential structure, and scale.

By Ekaterina Grishina, Stepan Kuznetsov, Askar Tsyganov, Ilya Ivanov, Daria Korovaitceva, Margarita Rusanova, Uliana Parkina, Alexander Derevyagin, Evgeny Frolov, Sergey Samsonov, Anton Lysenko

efficiency benchmarks

arXiv Machine Learning

Jun 2

BLISS: A Lightweight Bilevel Influence Scoring Method for Data Selection in Language Model Pretraining

arXiv:2510. 06048v4 Announce Type: replace Abstract: Effective data selection is essential for pretraining large language models (LLMs), enhancing efficiency and improving generalization to downstream tasks.

By Jie Hao, Rui Yu, Wei Zhang, Huixia Wang, Jie Xu, Mingrui Liu

llms benchmarks

arXiv Machine Learning

Jun 2

Tempora: Characterising the Time-Contingent Utility of Online Test-Time Adaptation

arXiv:2602. 06136v2 Announce Type: replace Abstract: Test-time adaptation (TTA) offers a compelling remedy for machine learning (ML) models that degrade under domain shifts, improving generalisation on-the-fly with only unlabelled samples.

By Sudarshan Sreeram, Young D. Kwon, Cecilia Mascolo

arXiv AI

Jun 30

Are LLMs Reliable Rankers? Rank Manipulation via Two-Stage Token Optimization

arXiv:2510. 06732v2 Announce Type: replace-cross Abstract: Large language models (LLMs) are increasingly used as rerankers in information retrieval, yet their ranking behavior can be steered by small, natural-sounding prompts.

By Tiancheng Xing, Jerry Li, Yixuan Du, Xiyang Hu

llms rag robotics safety

arXiv AI

6d ago

CAR: Query-Guided Confidence-Aware Reranking for Retrieval-Augmented Generation

arXiv:2605. 04495v2 Announce Type: replace-cross Abstract: Retrieval-augmented generation (RAG) relies on evidence ranking to determine what information is exposed to the generator, yet existing retrieval and reranking methods primarily estimate query--document relevance.

By Zhipeng Song, Yizhi Zhou, Xiangyu Kong, Jiulong Jiao, Xuezhou Ye, Chunqi Gao, Xueqing Shi, Yu Wang, Yuhang Zhou, Heng Qi

llms rag

arXiv Machine Learning

Jul 23

Statistical Inference for Rank Allocation in Low-Rank Adaptation

arXiv:2607. 20205v1 Announce Type: cross Abstract: Low-rank adaptation (LoRA) has become a widely used parameter-efficient fine-tuning method for large language models.

By Yihang Gao, Vincent Y. F. Tan

llms nlp fine-tuning

arXiv AI

Jun 26

The Capability Frontier: Benchmarks Miss 82% of Model Performance

arXiv:2606. 26836v1 Announce Type: new Abstract: Existing benchmarks typically report accuracy for a single model on a single run.

By Bradley Fowler, Ryan Smith, Daniel Thi Graviet, William Myers, Joshua Greaves, Narmeen Fatimah Oozeer, Ant\'ia Garc\'ia, Philip Quirke, Amirali Abdullah, Fazl Barez, Shriyash Kaustubh Upadhyay

llms agents benchmarks

arXiv AI

Jun 26

Estimating Uncertainty in Classifier Performance with Applications to Large Language Models and Nested Data

arXiv:2606. 26422v1 Announce Type: new Abstract: Researchers increasingly use text classification--supervised models or large language models--to measure constructs from natural language, providing metrics such as recall and precision as evidence of their validity.

By Kylie Anglin

llms nlp

Related stories