← Back to all news
arXiv Machine Learning By Jeffery Opoku, David Banahene

CleanScore: Black-Box Benchmark Audits with Negative Controls and Sensitivity Bounds

Read the original on arXiv Machine Learning →

CleanScore is a black‑box audit framework that uses only scored outputs to assess whether models have been exposed to benchmark questions. It creates a public form and two fresh, independently written forms for each question, reports an interval for the public‑form advantage, and employs a private negative‑control bank with a transport radius to separate exposure from normal form mismatch. In a registered audit of five open models on 200 GSM8K and 200 ARC‑Challenge items, CleanScore found no exposure‑consistent advantage, bounding surface‑form inflation below five points, while also demonstrating how leaked items can inflate accuracy on unseen paraphrases and how planted advantages can be partially detected even after rewriting. whyItMatters":"The study shows that CleanScore can detect and quantify exposure effects in benchmark models, providing a more nuanced understanding of model performance beyond simple accuracy scores."

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv Computation and Language
Sep 24

Beyond Overlap: Estimating the Causal Effect of Benchmark Exposure

LeakScale is an interventional framework that estimates the causal effect of benchmark exposure on model performance. By creating new executable tasks that require private, family‑specific information and controlling access to that information, LeakScale measures the control‑adjusted change in accuracy. Across 2,048 families, two model families, two domains, and 262,144 generations, exposure consistently improves accuracy by 7.17 to 27.31 percentage points, separating the question of whether contact occurred from how much a score depends on it.

By Divyansh Singh
benchmarks
More like this →
arXiv Computation and Language
6d ago

LeakScale: Estimating the Causal Effect of Benchmark Exposure

LeakScale is an interventional framework that estimates the causal effect of benchmark exposure on model performance. By creating new executable tasks that require private, family‑specific information not present in the public task, LeakScale controls access to that information and measures the resulting change in accuracy. Across 2,048 families, two model families, two domains, and 262,144 generations, exposure consistently improves accuracy, with gains ranging from +7.17 to +27.31 percentage points, thereby distinguishing whether benchmark contact occurred from how strongly a reported score depends on it.

By Divyansh Singh
benchmarks
More like this →
arXiv AI
Sep 1

Benchmark Contamination: A Taxonomy Organized by Defeated Mitigation

The paper introduces a new taxonomy for benchmark contamination that categorizes leakage by the mitigation it defeats—direct, derivative, temporal, distributional, and acquired—covering both training‑time and evaluation‑time scenarios. It proposes a four‑field disclosure protocol to record contamination status alongside benchmark scores, and provides a JSON schema, validator, and examples. An empirical study of 41 documents using a pre‑registered instrument shows limited reporting of contamination types and variable reliability, highlighting gaps in current disclosure practices.

By Johanna Angulo, V\'ictor Yeste, Hector Espinos-Morato
benchmarks
More like this →
arXiv Machine Learning
1d ago

When a Data Artifact Isn't a Shortcut: Causal Auditing of Synthetic RLVR Corpora

The paper audits whether synthetic distractors in RLVR corpora act as shortcuts for learning policies. A classifier using only surface statistics barely outperforms chance, and manual inspection reveals that code distractors are almost identical to correct answers. Experiments with a paraphrase‑matched control show no exploitation advantage for the unmodified data, indicating that the detectable artifact was not used by the policy.

By Esther Xin
llms
More like this →