arXiv Machine Learning
Jul 31
What Must a Fairness Audit Report When Demographic Data Is Incomplete?
arXiv:2506. 23033v5 Announce Type: replace Abstract: Fairness audits are a key component of responsible machine-learning deployment.
How Reliable are Fairness Audits with Unreliable Data?
arXiv:2506. 23033v2 Announce Type: replace Abstract: Fairness audits are a key component of responsible machine-learning deployment.
Related stories
arXiv AI
Sep 17
Pay Only for Disagreement: Certified No-Regression Verdicts for Model Updates with Matching Label-Complexity Bounds
The paper introduces DISCERN, a two-tier protocol for certifying that updates to production models do not increase risk. It first uses unlabeled data to detect benign updates based on disagreement rates, then selectively labels only disagreements through an anytime-valid confidence sequence. The method achieves finite-sample validity with label-complexity bounds of order ρ²/ε², demonstrating significant label savings and strong empirical performance across 14,000+ audit streams.
arXiv Statistics ML
2d ago
Tolerance-Based Fairness Auditing: Violation Certification and Sensitivity Screening
arXiv:2610.01005v1 Announce Type: new Abstract: As artificial intelligence is increasingly deployed, algorithmic unfairness has raised growing concerns and intensified demands for transparent fairnes...
arXiv Machine Learning
Sep 1
Audit Me If You Can: Query-Efficient Active Fairness Auditing of Black-Box LLMs
arXiv:2601.03087v2 Announce Type: replace Abstract: Large Language Models (LLMs) exhibit systematic biases across demographic groups. Auditing is proposed as an accountability tool for black-box LLM...
arXiv AI
Jun 3
The Reliability Gap in Benchmark Auditing: Distribution Shift and Scale as Failure Modes of Contamination Detection
arXiv:2606. 03305v1 Announce Type: new Abstract: Benchmark contamination, where evaluation examples appear in a model's training data, threatens the validity of LLM assessment.
arXiv Machine Learning
Sep 22
CleanScore: Black-Box Benchmark Audits with Negative Controls and Sensitivity Bounds
CleanScore is a black‑box audit framework that uses only scored outputs to assess whether models have been exposed to benchmark questions. It creates a public form and two fresh, independently written forms for each question, reports an interval for the public‑form advantage, and employs a private negative‑control bank with a transport radius to separate exposure from normal form mismatch. In a registered audit of five open models on 200 GSM8K and 200 ARC‑Challenge items, CleanScore found no exposure‑consistent advantage, bounding surface‑form inflation below five points, while also demonstrating how leaked items can inflate accuracy on unseen paraphrases and how planted advantages can be partially detected even after rewriting. whyItMatters":"The study shows that CleanScore can detect and quantify exposure effects in benchmark models, providing a more nuanced understanding of model performance beyond simple accuracy scores."
arXiv Machine Learning
Jun 16
Auditing Machine Unlearning: A Systematic Research on Whether Models Truly Forget
arXiv:2606. 16110v1 Announce Type: new Abstract: Machine unlearning has been extensively studied in response to growing privacy concerns and regulatory requirements.
arXiv Machine Learning
Sep 15
Certifying Model Upgrades with Slice-Wise Non-Regression and Incumbent Fallback
arXiv:2609.13714v1 Announce Type: new Abstract: An updated model can improve an aggregate metric while degrading a slice that matters to a downstream user. We study checkpoint selection subject to no...
arXiv AI
3d ago
Efficient Active Auditing of Multi-Group Fairness with Bias Probes
arXiv:2609.40034v1 Announce Type: cross Abstract: Over the past decade, Machine Learning (ML) has been trained under dual objectives: minimizing prediction error via Empirical Risk Minimization (ERM)...
arXiv AI
Aug 18
What Do Compliance Detectors Read? An Audit of Activation Probes and Guard Models
arXiv:2608. 16852v1 Announce Type: new Abstract: Regulatory compliance monitoring in deployed language models is increasingly implemented as a legal and audit control, checking model outputs against written rules spanning data protection, healthcare, financial regulation, and platform policy.
arXiv AI
Jul 2
FLAT: Revealing Hidden Latent-Conditioned Backdoor Failures in Federated Learning
arXiv:2508. 04064v2 Announce Type: replace-cross Abstract: Horizontal federated learning (HFL) backdoor audits often summarize model behavior through clean accuracy (CA), mean attack success rate (ASR), or a single known-trigger test.
arXiv Machine Learning
Jul 24
Finite-Sample Coverage Audits for High-Recall Candidate Generation: Certification and Learning-Theoretic Design
arXiv:2607. 21480v1 Announce Type: new Abstract: An initial high-recall stage in an empirical pipeline decides which items pass to later review, labelling, or modelling, and relevant items it misses are lost to every subsequent stage.