arXiv Machine Learning September 15, 2026 By Ricardo Fitas

The geometry of AI validation: From structural blindness to reusable audits

Read the original on arXiv Machine Learning →

The paper investigates how AI systems that perform best‑of‑n search require different validation strategies as the search width changes. It shows that auditing only small search widths leaves a gap in reliability estimates for larger widths, and proposes retaining candidate ranks and truth labels to estimate reliability across all widths up to N. The authors derive theoretical bounds on the minimax mean‑squared error, design procedures that achieve these bounds, and demonstrate that a shared audit can significantly reduce maximum error across many widths in practical CodeRM pools.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv Statistics ML

Aug 25

The geometry of AI validation: Exact certification limits for iid best-of-N search

arXiv:2608.21496v1 Announce Type: cross Abstract: AI systems increasingly generate alternatives, inspect evidence, and deploy a selected output. Validation is therefore target-relative: evidence cert...

By Ricardo Fitas

Hugging Face Trending Papers

Aug 18

Judge, Retrieve, or Abstain: Uncertainty-Guarded LLM Judging with Provable Risk Guarantees

The paper introduces a risk‑controlled framework for using large language models (LLMs) as judges in tasks without reference answers. By calibrating uncertainty thresholds on a held‑out set, the method ensures that the false discovery rate of accepted verdicts stays below a user‑specified level α with high probability, using finite‑sample Clopper–Pearson intervals. When the parametric judge lacks confidence, the instance is routed to a retrieval‑augmented mode with a second calibrated threshold, preserving the error guarantee while achieving higher coverage than single‑mode baselines.

llms rag benchmarks safety

arXiv AI

Aug 11

AI Evaluation Should Measure Verification Cost, Not Correctness Alone

arXiv:2608. 08709v1 Announce Type: new Abstract: The reliability of AI generative models is typically measured by output correctness, yet in practice it depends on the effort required to verify those outputs.

By Viviana Crescitelli, Generoso Immediato, Fabio Persia, Stefania Costantini

multimodal benchmarks safety

arXiv Machine Learning

Aug 4

When May a Model Replace the Experiment? Audits, Licenses, and the Price of Trust in Surrogate-Driven Design

arXiv:2608. 01378v1 Announce Type: new Abstract: Design campaigns in chemistry, materials science, and machine learning share a bottleneck: determining how good a candidate truly is requires an expensive evaluation - an experiment, a first-principles simulation, or a full training run.

By Shuangxiu (Max), Ma (Zachary), Wenhe (Zachary), Zhao

safety

arXiv AI

Aug 7

SearchAuditor: Auditing and Attributing Failures in Long-Horizon Search Agents

arXiv:2608. 05212v1 Announce Type: new Abstract: Deep search agents tackle challenging questions through long-horizon web interactions, a process that is both complex and fragile: small reasoning errors may propagate through long, noisy trajectories into fluent but incorrect answers.

By Zhixiang Liang, Yifei Liu, Yidan Huang, Haozhe Zhao, Beichen Huang, Jiaqi Wang, Nan Duan, Qiong Cao

llms agents benchmarks

arXiv Machine Learning

Jun 29

CELEUS: Certifiable and Efficient LLM Evaluation via E-Processes

arXiv:2606. 20820v2 Announce Type: replace Abstract: Can we trust evaluation scores to capture an LLM's true real-world performance?

By Zhijian Zhou, Zesheng Ye, Zhaorun Chen, Bo Li, Feng Liu

llms