arXiv AI By George Koomullil (Ascendr, PBC)

Counting Under Disagreement: Dollar-Weighted Measurement, Selective Machine Labelling and Budgeted Human Review from Disagreeing AI Readers of Administrative Text

Read the original on arXiv AI →

The paper investigates AI‑assisted measurement for administrative records where a single text label applies to many records and the target is a monetary total. Using 121,070 grant‑purpose texts from US tax filings, the authors compare rule‑based and language‑model classifiers, evaluate fixed‑predictor and cross‑fitted estimators under Poisson sampling, and test dollar‑weighted error rules and a two‑phase review policy. Results show that a calibrated two‑phase policy can reduce RMSE by 28% with a limited budget, while independent human readers shift totals by billions and institutional intervals under‑cover.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv Computation and Language
Oct 27

From Self-Confidence to Statistical Accuracy: Scaling Human-in-the-Loop Evaluation with Direct and LLM-Assisted Auditing

arXiv:2610.35698v1 Announce Type: new Abstract: For large language models (LLMs) and multimodal large language models (MLLMs), model-reported self-confidence is often miscalibrated, yet it may still...

By Harine Choi (Texas A&M Transportation Institute, Texas A&M University System), Chenchen Kuai (Zachry Department of Civil and Environmental Engineering, Texas A&M University), Zihao Li (Department of Civil, Construction and Environmental Engineering, Marquette University), Jiwan Jiang (Zachry Department of Civil and Environmental Engineering, Texas A&M University), Yunlong Zhang (Zachry Department of Civil and Environmental Engineering, Texas A&M University), Zhiwen Fan (Department of Electrical and Computer Engineering, Texas A&M University), Zhengzhong Tu (Department of Computer Science and Engineering, Texas A&M University), Yang Zhou (Zachry Department of Civil and Environmental Engineering, Texas A&M University)
arXiv Machine Learning
Oct 19

Sample, Mix, or Verify? Allocating Test-Time Compute under Imperfect Verifiers

The paper introduces SPLIT, a method that optimally allocates test‑time compute among sampling, model selection, and verification for language models and agents. Using an exact probability formula that accounts for task difficulty, shared blind spots, and verifier correlations, SPLIT is fitted via empirical Bayes on a small labeled pilot and then used to plan budgets for unseen tasks. Across seven diverse benchmarks, 26 models, and six verifier types, SPLIT achieves near‑optimal performance, outperforming rule‑based baselines by up to 2.5 percentage points and providing insights into when additional LLM judges reduce precision.

By Srinivasan Iyengar
arXiv AI
Aug 20

Debiased Inference for AI-Generated Data without Gold-Standard Labels: Identification via Multiple Imperfect Measurements

The paper introduces Debiased Inference with Multiple Imperfect Measurements (DMM), a framework that uses several error‑prone AI measurements to perform valid downstream statistical inference without requiring costly gold‑standard labels. By assuming conditional independence of the measurements given the true label and unit‑level features, DMM leverages CP decomposition and semiparametric theory to prove consistency and asymptotic normality of its estimator. Simulations demonstrate that DMM yields valid inference and can improve efficiency when additional imperfect measurements are available, and the authors provide diagnostics for the key independence assumption.

By Naoki Egami, Sooahn Shin