The paper audits vision‑language models (CLIP) for gender bias using 1,500 artworks from the Metropolitan Museum of Art, focusing on zero‑shot logit differences for prompts like "masterpiece," "quality," and "influence." Unadjusted results show no significant gender effect and statistical equivalence across models, but high residual variance suggests that global zero‑shot metrics are largely noise‑dominated and may miss fine‑grained biases. The study underscores the need for multivariate confound control, equivalence testing, and provenance auditing when evaluating AI fairness in cultural heritage data.
arXiv:2607. 28934v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly involved in the distribution of scarce resources, raising concerns about biased allocations based on characteristics like race and gender.
The paper examines whether removing declared language fields from de‑identified résumés eliminates demographic leakage in large language models. By keeping language attributes identical and varying only unstructured prose across five ethnocultural groups and three cue‑salience levels, the authors find that non‑language text still allows target‑group recovery (average 0.757, reaching 1.000 under high salience). They also show that evaluation design—such as allowing or forbidding ties—dramatically affects LLM‑as‑a‑judge outcomes, underscoring the importance of evaluation protocol in bias audits.
The study examines how the design of audit questions influences perceived bias in large language models (LLMs). Using 40,726 requests across five models and three domains—hiring, lending, and medical triage—the authors find that demographic bias effects are not replicated when the audit is standardized. Instead, the audit’s construction, such as question phrasing and ordering, has a stronger impact on model responses than applicant demographics.
The paper introduces an audit layer for vision‑language model benchmarks that reuses per‑question artifacts from evaluation runs to assess the benchmark itself. Six detectors analyze saved outputs, and an additional text‑only rerun removes visual inputs to produce per‑question reports and aggregate statistics, including permutation‑null excess and interval estimates. Applying this method to the MMStar benchmark reveals that 20.2% of its questions show no accuracy gain from images, with a small but significant excess in mathematics and science categories, and identifies questions where all models chose the same non‑gold option.
arXiv:2608.29590v1 Announce Type: new Abstract: We propose a societal bias evaluation method for large vision-language models (LVLMs) in the era of strong safety guardrails. Existing benchmarks rely...
arXiv:2609.24885v1 Announce Type: new Abstract: When a language model answers from a curated corpus via graph-based retrieval, a large grounding uplift does not establish reasoning over the retrieved...
arXiv:2608.21415v1 Announce Type: cross Abstract: Large Vision-Language Models (LVLMs) have achieved remarkable performance across a wide range of tasks; however, they often inherit social biases fro...
The paper investigates how large language models (LLMs) from OpenAI and Anthropic exhibit probability bias when deciding whether to trust information from reporters of different racial descriptions. Using a two‑urn Bayesian game and 6,600 sessions, the authors find that models often refuse to assess a reporter’s credibility—especially for Black‑described reporters—yet when they do assess, Black reporters receive slightly higher trust. The bias is subtle, growing with reasoning effort, and is not detected by standard audits that compare average trust or only assessed cases.
arXiv:2604. 09945v2 Announce Type: replace-cross Abstract: The rapid adoption of large vision-language models (LVLMs) in recent years has been accompanied by growing fairness concerns due to their propensity to reinforce harmful societal stereotypes.
FairLens is a benchmark and evaluation framework that measures fairness and validity of vision‑language models (VLMs) in high‑stakes domains such as hiring, legal, and healthcare. It uses over 100,000 face‑image and question pairs covering gender, race, and age, and assesses responses through demographic parity, soundness, demographic association, and bias in free‑text generation. The study finds that VLMs often make unwarranted inferences from faces rather than abstaining, especially in legal and healthcare contexts, and that small parity gaps can still hide unsafe treatment across groups.
arXiv:2608. 06908v1 Announce Type: cross Abstract: We propose Zero-phase Component Analysis (ZCA) whitening as a geometric pre-processing step for the Word Embedding Association Test (WEAT).
arXiv:2609.15100v1 Announce Type: cross Abstract: An image tool can change its underlying generator while retaining its public name, making version attribution from online posts ambiguous. We study t...