arXiv Machine Learning September 3, 2026 By Vahid Reza Khazaie, Ahmed Y. Radwan, Shaina Raza

FairLens: Benchmarking Fairness in Vision-Language Models for High-Stakes Decision-Making

Read the original on arXiv Machine Learning →

FairLens is a benchmark and evaluation framework that measures fairness and validity of vision‑language models (VLMs) in high‑stakes domains such as hiring, legal, and healthcare. It uses over 100,000 face‑image and question pairs covering gender, race, and age, and assesses responses through demographic parity, soundness, demographic association, and bias in free‑text generation. The study finds that VLMs often make unwarranted inferences from faces rather than abstaining, especially in legal and healthcare contexts, and that small parity gaps can still hide unsafe treatment across groups.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

arXiv AI

Jul 24

Unlearning Under Imbalance: Benchmarking Fairness in Multimodal LLM Unlearning

arXiv:2607. 21300v1 Announce Type: cross Abstract: Machine unlearning has emerged as a tool for removing personal data from trained models to comply with recent AI regulations.

By Lorenzo Orsingher, Thomas De Min, Massimiliano Mancini, Davide Talon, Elisa Ricci

llms nlp fine-tuning multimodal benchmarks safety

arXiv AI

Aug 25

Mitigating Bias in Large Vision-Language Models via Counterfactual Ensemble Decoding

arXiv:2608.21415v1 Announce Type: cross Abstract: Large Vision-Language Models (LVLMs) have achieved remarkable performance across a wide range of tasks; however, they often inherit social biases fro...

By Yisong Xiao, Aishan Liu, Yongxin Huang, Zonghao Ying, Shiji Zhao, Tianlin Li, Yong Han, Jian Yang, Xianglong Liu

llms multimodal benchmarks safety

arXiv Computer Vision

Sep 1

Guardrail-Agnostic Societal Bias Evaluation in Large Vision-Language Models

arXiv:2608.29590v1 Announce Type: new Abstract: We propose a societal bias evaluation method for large vision-language models (LVLMs) in the era of strong safety guardrails. Existing benchmarks rely...

By Yusuke Hirota, Michael Ross Boone, Arun George Zachariah, Jibin Rajan Varghese, Yu-Chiang Frank Wang, Boyi Li, Ryo Hachiuma

llms multimodal benchmarks safety

arXiv Machine Learning

Jul 1

WIDER-FAIR: An Annotated Version of the WIDER-FACE Dataset for Fairness Evaluation

arXiv:2606. 31704v1 Announce Type: cross Abstract: The deployment of face detection models in real-world applications raises important fairness concerns, as these systems may showcase performance disparities across demographic groups.

By Maxime Moussi, Beno\^it Ronval, Siegfried Nijssen, F\'elicien Schiltz

rag benchmarks safety

Hugging Face Trending Papers

Aug 10

CIFA: Contextual-Intersectional Fairness Auditing for Hidden Subgroup Discovery in Face Analysis

Fairness evaluation in computer vision commonly relies on aggregate accuracy and demographic subgroup analysis. However, visual models are also sensitive to contextual factors such as illumination, blur, image quality, facial accessories, and appearance attributes.

computer-vision safety

arXiv Machine Learning

Jun 9

Exposing Hidden Biases in Text-to-Image Models via Automated Prompt Search

arXiv:2512. 08724v3 Announce Type: replace Abstract: Text-to-image (TTI) diffusion models have achieved remarkable visual quality, yet they have been repeatedly shown to exhibit social biases across sensitive attributes such as gender, race and age.

By Manos Plitsis, Giorgos Bouritsas, Vassilis Katsouros, Yannis Panagakis

llms diffusion benchmarks safety