arXiv Statistics ML
Not All Mislabels Are Equal: Efficient Task-Aware Auditing with Confidence Sequences
Read the original on arXiv Statistics ML →The Flow has not summarised this story yet — read it at arXiv Statistics ML.
Related stories
arXiv Machine Learning
Sep 23
Targeted Review for AI-Assisted Biodiversity Surveys: Active Continuous-Score Occupancy Modeling
The paper introduces ACORN, a method that blends machine‑learning predictions with occupancy models to guide ecologists in selecting which samples to review. By strategically choosing the most informative labels, ACORN achieves ecological conclusions nearly identical to fully human‑labeled data while dramatically reducing the number of expert reviews needed. The approach is evaluated on camera‑trap and bioacoustic datasets, demonstrating its effectiveness across real‑world biodiversity surveys.
arXiv Machine Learning
Sep 15
The geometry of AI validation: From structural blindness to reusable audits
The paper investigates how AI systems that perform best‑of‑n search require different validation strategies as the search width changes. It shows that auditing only small search widths leaves a gap in reliability estimates for larger widths, and proposes retaining candidate ranks and truth labels to estimate reliability across all widths up to N. The authors derive theoretical bounds on the minimax mean‑squared error, design procedures that achieve these bounds, and demonstrate that a shared audit can significantly reduce maximum error across many widths in practical CodeRM pools.
Hugging Face Trending Papers
Aug 18
Judge, Retrieve, or Abstain: Uncertainty-Guarded LLM Judging with Provable Risk Guarantees
The paper introduces a risk‑controlled framework for using large language models (LLMs) as judges in tasks without reference answers. By calibrating uncertainty thresholds on a held‑out set, the method ensures that the false discovery rate of accepted verdicts stays below a user‑specified level α with high probability, using finite‑sample Clopper–Pearson intervals. When the parametric judge lacks confidence, the instance is routed to a retrieval‑augmented mode with a second calibrated threshold, preserving the error guarantee while achieving higher coverage than single‑mode baselines.
arXiv Machine Learning
Sep 24
Active Learning for Biodiversity Monitoring: From Label Efficiency to Reliable Ecological Inference
The paper discusses how active learning (AL) can alleviate the expert annotation bottleneck in biodiversity monitoring by selecting the most informative samples under a fixed budget. It highlights that while AL reduces labeling effort, its non-random sample selection complicates model validation, calibration, and ecological inference, issues often overlooked in current studies. The authors review existing AL research across acoustic and image data, identify gaps such as limited species coverage and lack of real-world deployments, and propose a tutorial framework and roadmap for developing AL methods that support efficient training, reliable validation, and trustworthy ecological conclusions.
arXiv Computation and Language
Oct 27
From Self-Confidence to Statistical Accuracy: Scaling Human-in-the-Loop Evaluation with Direct and LLM-Assisted Auditing
arXiv:2610.35698v1 Announce Type: new Abstract: For large language models (LLMs) and multimodal large language models (MLLMs), model-reported self-confidence is often miscalibrated, yet it may still...
arXiv AI
Oct 1
Auditing Agent Actions through Query-Conditioned Attribution
arXiv:2609.33676v2 Announce Type: replace Abstract: LLM agents increasingly take consequential actions through interactions with users, policies, and external tools. Auditing these agents requires au...