arXiv:2607. 08522v1 Announce Type: new Abstract: The inherent rigidity of fixed-size benchmarks makes them an inefficient tool for model evaluation.
arXiv:2602. 06136v2 Announce Type: replace Abstract: Test-time adaptation (TTA) offers a compelling remedy for machine learning (ML) models that degrade under domain shifts, improving generalisation on-the-fly with only unlabelled samples.
arXiv:2608. 14425v1 Announce Type: new Abstract: LLM evaluations often use fixed sampling budgets, testing every item the same number of times even after estimates are precise.
The paper introduces VTC-Bench, a five‑domain benchmark designed to evaluate multiple outputs from large language models (LLMs) by measuring Validated Task Coverage (VTC). VTC quantifies how many distinct, useful results are produced within a set number of attempts, using real‑data tasks that allow automatic, reproducible checks of output quality and task‑relevant distinctness without relying on model‑based judges. Experiments show that models which perform best on single‑draw quality do not always achieve the highest coverage, and simple output‑variation metrics fail to capture task‑relevant diversity, highlighting the importance of evaluating finite candidate sets directly.
arXiv:2606. 24020v1 Announce Type: new Abstract: A modern model release reports scores on 40+ benchmarks and the same evaluations were run many more times before it: to track training progress, compare design choices, and select the checkpoint for the release.
arXiv:2601.13885v2 Announce Type: replace-cross Abstract: Computerized Adaptive Testing (CAT) has proven effective for efficient LLM evaluation on multiple-choice benchmarks, but modern LLM evaluatio...
arXiv:2606. 15237v1 Announce Type: cross Abstract: Ensemble classifiers are predictive models that combine the results of simpler base models, often by majority vote.
arXiv:2609.13714v1 Announce Type: new Abstract: An updated model can improve an aggregate metric while degrading a slice that matters to a downstream user. We study checkpoint selection subject to no...
arXiv:2606. 04525v1 Announce Type: cross Abstract: Progress in genomic foundation models is difficult to assess due to fragmented benchmarks, incompatible evaluation protocols, and task-specific reporting.
arXiv:2608.27704v1 Announce Type: new Abstract: When machine learning classifiers are retrained, inputs correctly classified by the previous model version may be misclassified by the updated version,...
The paper examines how the choice of anchor model in LLM-as-a-judge evaluations affects reliability. By testing 22 anchors on the Arena-Hard-v2.0 dataset, it shows that extreme anchors (best or worst performers) are poor choices, reducing correlation with human rankings. The study quantifies the anchor effect size, compares it to judge model selection, and offers guidelines and power‑analysis recommendations for more reliable benchmark design.
The paper investigates how test‑time computation can enhance language models and at what cost, introducing the SELF‑POT benchmark to evaluate this across competition mathematics, competitive programming, and agentic workflows. SELF‑POT separates candidate coverage from final accuracy, tracks correctness transitions under revision, and measures protocol completion alongside task success. Using a unified budget rule, the study compares direct inference, parallel sampling, and self‑revision across five low‑cost reasoning models, revealing that selection rules and failure handling significantly influence gains and cost savings.
arXiv:2610.00993v1 Announce Type: new Abstract: Before a machine learning model ships, it often has to pass a suite of automated tests. Requiring every test to pass looks safe, yet it can reject many...