arXiv Machine Learning

A Unified Framework for Probability Estimation of Data-Dependent Events

The paper introduces a unified framework for estimating the probability that an unknown distribution assigns to a data‑dependent event defined by a sample and a known mapping. It analyzes two classical estimators—the empirical plug‑in estimator and the leave‑one‑out (Good–Turing) estimator—showing how their errors relate to generalization gaps and stability under sample deletion. The framework is applied to error estimation for predictors that combine a base classifier with memorized exceptions, leading to new bounds based on VC dimension and mutual information for adaptive missing mass estimation.

arXiv Machine Learning
Oct 7

Assumption-lean logistic regression with missing covariates

arXiv:2610.07292v1 Announce Type: cross Abstract: Missing covariates are frequently encountered in supervised learning problems, and classical methods for estimation using such data use carefully cho...

By Jyotishka Ray Choudhury, Kabir Aladin Verchand, Richard J. Samworth, Ashwin Pananjady
arXiv Machine Learning
Oct 22

Approximate Solomonoff Induction: On the Robustness of Learned Universal Predictors

The paper investigates how well practical machine‑learning algorithms approximate Solomonoff’s universal predictor when trained by minimizing log‑loss over diverse datasets. It establishes tight bounds on the worst‑case increase in prediction error caused by approximating the Bayesian mixture predictor, showing that while the excess error can be large in low‑prior environments, it remains bounded on average and under distribution shifts. The authors also extend their analysis to finite‑sample settings relevant to neural networks and Transformers.

By Qian Li, Shang-Hua Teng
arXiv Machine Learning
Jun 9

Partial Identification under Missing Data Using Weak Shadow Variables from Pretrained Models

arXiv:2602. 16061v2 Announce Type: replace-cross Abstract: Estimating population quantities such as mean outcomes from user feedback is fundamental to platform evaluation and social science, yet feedback is often missing not at random (MNAR): users with stronger opinions are more likely to respond, so standard estimators are biased and the estimand is not identified without additional assumptions.

By Hongyu Chen, David Simchi-Levi, Ruoxuan Xiong
arXiv Machine Learning
Sep 18

Next-token functional estimation

The paper introduces a leave‑a‑window‑out estimator for next‑token functionals, such as the surprise probability and test error, in sequences of random variables. By deleting a window of length τ after each index, the estimator generalizes leave‑one‑out and achieves parametric error decay for stationary β‑mixing processes that admit a Marton coupling. The authors provide both upper bounds and a minimax lower bound for the surprise probability, and demonstrate through simulations that their method outperforms traditional baselines on Markov, moving‑average, and autoregressive processes.

By Milind Nakul, Vidya Muthukumar, Ashwin Pananjady
Hugging Face Trending Papers
Sep 17

Next-token functional estimation

Suppose we observe the first $n$ points of a sequence of random variables having length $n+1$, and wish to estimate a functional of the unobserved final point and the empirical measure of the $n$ obse...

arXiv Machine Learning
Sep 3

Bayes-Optimal BER and AUC: Estimation and Evaluation of Estimators

The paper introduces soft‑label‑based estimators for the Bayes‑optimal balanced error rate (BER) and area under the ROC curve (AUC), extending from a clean setting with known class priors to a realistic scenario with unknown priors and corrupted soft labels. It also adapts the FeeBee evaluation framework to assess these estimators without needing the true optimum, providing practical evaluation scores for any estimator of optimal BER or AUC. Experiments on synthetic and real datasets confirm the effectiveness of both the estimators and the evaluation method.

By Ryota Ushio, Takashi Ishida, Masashi Sugiyama
arXiv Machine Learning
Oct 16

Computational and Statistical Hardness of Calibration Distance

The paper investigates the computational and statistical challenges of the calibration distance, a key metric for measuring miscalibration in probabilistic predictors. It presents an efficient algorithm for exact computation when the distribution has a uniform marginal and noiseless labels, and shows that the problem becomes NP‑hard if either assumption is dropped. The authors also provide a polynomial‑time approximation scheme for the general case and establish tight sample complexity bounds for estimating the calibration distance from data.

By Mingda Qiao
arXiv Statistics ML
Nov 9

Instance-Optimal Estimation with Multiple LLM Judges on a Budget

The paper addresses how to allocate a fixed evaluation budget across heterogeneous large language model judges to estimate scores for multiple prompt-response pairs most accurately. It introduces an inverse‑variance weighted estimator (IVWE) and an adaptive algorithm, EST‑IVWE, that uses optimistic variance estimates to approximate the oracle allocation. The authors prove that EST‑IVWE achieves near‑oracle error rates and establish a local minimax lower bound, demonstrating asymptotic instance‑optimality up to logarithmic factors, and validate the method on synthetic data and HelpSteer2 evaluations.

By Junghyun Lee, Sanghwa Kim, Yassir Jedra, Alexandre Prouti\`ere, Se-Young Yun
arXiv AI
Aug 20

Debiased Inference for AI-Generated Data without Gold-Standard Labels: Identification via Multiple Imperfect Measurements

The paper introduces Debiased Inference with Multiple Imperfect Measurements (DMM), a framework that uses several error‑prone AI measurements to perform valid downstream statistical inference without requiring costly gold‑standard labels. By assuming conditional independence of the measurements given the true label and unit‑level features, DMM leverages CP decomposition and semiparametric theory to prove consistency and asymptotic normality of its estimator. Simulations demonstrate that DMM yields valid inference and can improve efficiency when additional imperfect measurements are available, and the authors provide diagnostics for the key independence assumption.

By Naoki Egami, Sooahn Shin