arXiv AI
Aug 19

Position: Fairness Failure in Generative Models is an Evaluation Problem

The paper argues that fairness failures in generative models arise mainly from inadequate evaluation practices, making fairness findings hard to compare or use for deployment. It diagnoses common empirical and conceptual shortcomings in current methods and calls for a move toward standardized, generative‑specific evaluation. The authors introduce Fairness Cards, a minimal reporting artifact that explicitly documents evaluation choices—such as prompt families, counterfactual protocols, metrics, and refusal handling—to improve reproducibility, comparability, and accountability.

By Mariia Vladimirova, Jean-Yves Franceschi, Thibaut Issenhuth
arXiv Machine Learning
Oct 2

Fairness Theatre: Evaluating Post-Hoc Fairness Interventions in Vendor-Controlled Early Warning Systems

The paper examines how public institutions, particularly a Canadian college, can address inequities in proprietary Early Warning Systems (EWS) when they cannot modify the underlying AI models. By testing six post‑hoc fairness interventions on a research EWS under realistic procurement constraints, the authors compare fairness, accuracy, and demographic disparities, and introduce error‑type profiling to track how interventions shift false positives and false negatives. The results show that while interventions can redistribute disparities, they do not consistently reduce them; some favor already‑advantaged groups, leaving small, marginalized groups poorly served, illustrating the limits of fairness work under vendor‑controlled systems.

By Kelly McConvey, Angelina Zhai, Rebecca Li, Shion Guha
arXiv Machine Learning
Jun 2

Beyond Procedure: Substantive Fairness in Conformal Prediction

arXiv:2602. 16794v2 Announce Type: replace-cross Abstract: Conformal prediction (CP) offers distribution-free uncertainty quantification for machine learning models, yet its interplay with fairness in downstream decision-making remains underexplored.

By Pengqi Liu, Zijun Yu, Mouloud Belbahri, Arthur Charpentier, Masoud Asgharian, Jesse C. Cresswell