arXiv:2608. 02985v1 Announce Type: new Abstract: The standard check for contamination in LLM backtests is simple: compare scores before and after the training cutoff.
By Zeyu Zhang, Bradly C. Stadie
arXiv:2606. 03305v1 Announce Type: new Abstract: Benchmark contamination, where evaluation examples appear in a model's training data, threatens the validity of LLM assessment.
By Wojciech Zarzecki, Jan Dubi\'nski, Sebastian Cygert
The paper argues that prediction‑based certifications—such as accuracy, calibration, and conformal coverage—are insufficient to guarantee trustworthy AI. It proves a separation theorem showing that a model can appear reliable under all prediction‑side certificates yet differ arbitrarily in explanation fidelity and deployment behaviour. The authors propose a competence envelope framework that combines both prediction and explanation certification to detect such hidden failures.
By Nataliya Shakhovska, Ivan Izonin, Stergios-Aristoteles Mitoulis
arXiv:2608. 02786v1 Announce Type: new Abstract: AI systems can fail silently.
By Priyanka Bajaj (Independent Researcher)
The paper introduces a new taxonomy for benchmark contamination that categorizes leakage by the mitigation it defeats—direct, derivative, temporal, distributional, and acquired—covering both training‑time and evaluation‑time scenarios. It proposes a four‑field disclosure protocol to record contamination status alongside benchmark scores, and provides a JSON schema, validator, and examples. An empirical study of 41 documents using a pre‑registered instrument shows limited reporting of contamination types and variable reliability, highlighting gaps in current disclosure practices.
By Johanna Angulo, V\'ictor Yeste, Hector Espinos-Morato
The study audited ten different classifiers—including linear, tree‑ensemble, neural, glass‑box, and tabular foundation models—on national health survey data to predict myocardial infarction. By systematically removing features that could cause target leakage, the authors found that all models’ AUROC scores collapsed into a narrow band, indicating that reported high accuracy in prior work was largely due to leakage rather than model sophistication. The glass‑box explainable boosting machine performed comparably to other models while being much faster, and the authors demonstrated that fairness, calibration, and uncertainty can be audited and repaired without sacrificing performance.
By Raad Bin Tareaf, Murad Al-Rajab, Samia Loucif, Samer Ellaham, Cedric Schmitz