arXiv Statistics ML
What Fixed-Rollout pass@k Evaluations Can Identify
Read the original on arXiv Statistics ML →The Flow has not summarised this story yet — read it at arXiv Statistics ML.
Related stories
Hugging Face Trending Papers
Sep 8
What Fixed-Rollout pass@k Evaluations Can Identify
Repeated-sampling evaluations increasingly extrapolate pass@k far beyond the number n of samples collected per problem. We show that, in the pooled/random-task conditional-Binomial model, fixed-n succ...
Hugging Face Trending Papers
Sep 24
Sharp Limits for Honest Uncertainty in Hard-Budget Repeated Evaluation
Repeated evaluation can estimate a benchmark score accurately while still requiring replication to certify narrow uncertainty. We characterize that requirement on a fixed grid of $M$ tasks with $L$ binary paths per task under the hard budget $(M+t)K$, where each path costs at most $K$ responses or episodes.
arXiv AI
6d ago
Sharp Limits for Honest Uncertainty in Hard-Budget Repeated Evaluation
arXiv:2609. 29140v1 Announce Type: new Abstract: Repeated evaluation can estimate a benchmark score accurately while still requiring replication to certify narrow uncertainty.
arXiv AI
Jun 9
Prescriptive Scaling Reveals the Evolution of Language Model Capabilities
arXiv:2602. 15327v2 Announce Type: replace-cross Abstract: Machine learning model performance improvements tend to arise from competition and application.
arXiv Machine Learning
Aug 18
Certifying Compressed Language Models: An Audit and a Statistical Toolkit
arXiv:2608. 15046v1 Announce Type: new Abstract: A fraction of a point of benchmark accuracy is the usual evidence that a compressed model is equivalent to its original.
arXiv AI
Sep 17
Confidence Comes from Experience: Experiential Confidence Estimation from Reasoning to Agents
The paper introduces XConf, an experiential confidence estimator that augments a language model’s current inference with a record of its past graded episodes. By recalling similar past tasks and reflecting on past outcomes, XConf generates confidence scores without accessing logits or updating weights, achieving superior discrimination and calibration across diverse benchmarks. The method demonstrates significant gains in selective prediction, improving success rates on agent tasks by up to 8.7 points.