The paper investigates how large language and visual‑language models used in autonomous vehicles inherit human driver biases when deciding whether to yield to pedestrians. It introduces two new bias‑testing methods—All Else Being Equal and Self‑Consistency tests—to evaluate these models. Results reveal that the models’ yielding decisions are influenced by pedestrian attributes such as gender, ethnicity, religion, disability, age, skin tone, and socio‑economic status, with varying patterns across models.
By Irem Yoldas, Martim Brand\~ao, Jie Zhang, Odinaldo Rodrigues
FairLens is a benchmark and evaluation framework that measures fairness and validity of vision‑language models (VLMs) in high‑stakes domains such as hiring, legal, and healthcare. It uses over 100,000 face‑image and question pairs covering gender, race, and age, and assesses responses through demographic parity, soundness, demographic association, and bias in free‑text generation. The study finds that VLMs often make unwarranted inferences from faces rather than abstaining, especially in legal and healthcare contexts, and that small parity gaps can still hide unsafe treatment across groups.
By Vahid Reza Khazaie, Ahmed Y. Radwan, Shaina Raza
arXiv:2503.00610v2 Announce Type: replace-cross
Abstract: Understanding how people perceive urban environments is essential for inclusive planning, yet conventional surveys are costly and difficult t...
By Ciro Beneduce, Bruno Lepri, Massimiliano Luca
arXiv:2608.29590v1 Announce Type: new
Abstract: We propose a societal bias evaluation method for large vision-language models (LVLMs) in the era of strong safety guardrails. Existing benchmarks rely...
By Yusuke Hirota, Michael Ross Boone, Arun George Zachariah, Jibin Rajan Varghese, Yu-Chiang Frank Wang, Boyi Li, Ryo Hachiuma
arXiv:2606. 11016v1 Announce Type: new Abstract: We ask whether large language models (LLMs) merely imitate rationales when choosing between two options, or whether their choices reflect a systematic underlying decision structure.
By Gabriel Freedman, Francesca Toni
arXiv:2604. 01925v2 Announce Type: replace-cross Abstract: Large Language Models increasingly suppress biased outputs when demographic identity is stated explicitly, yet may still exhibit implicit biases when identity is conveyed indirectly.
By Bhaskara Hanuma Vedula, Darshan Anghan, Ishita Goyal, Ponnurangam Kumaraguru, Abhijnan Chakraborty
arXiv:2606. 17362v1 Announce Type: cross Abstract: Autonomous driving has shifted towards end-to-end policy learning, where reliable, interpretable policy evaluation is a fundamental challenge as driving quality is highly context-dependent.
By Xinglong Sun, Kevin Xie, Jenny Schmalfuss, Despoina Paschalidou, Xiuming Zhang, Sanja Fidler, Kashyap Chitta, Jose M. Alvarez
Melange is a synthetic visual bias benchmark designed to evaluate social biases in multi‑modal large language models (MLLMs). It extends the language‑only BBQ dataset by generating controlled visual scenarios that clearly depict target attributes, and validates images through automated VQA screening and human inspection. Using Melange, 23 open‑source MLLMs were tested in text‑only, image‑only, and image‑with‑context settings, revealing that switching from text to vision reduces fairness by an average of 0.291 and that models often choose stereotypical responses when biased answers are made.
By Razieh Rezaei, Ehsaneddin Asgary, Masoud Jalili Sabet
LightEMMA is a longitudinal evaluation framework that tests the autonomous driving performance of vision‑language models (VLMs) without fine‑tuning or prompt engineering. Using this protocol, the authors evaluated 15 VLMs from five major families on the nuScenes prediction benchmark and found that larger, more capable models do not consistently outperform earlier generations. The study identifies common failure modes such as overreliance on historical actions and difficulty reconciling conflicting visual cues, underscoring the need for domain‑specific adaptation to enhance VLM safety in autonomous driving.
By Zhijie Qiao, Haowei Li, Zhong Cao, Henry X. Liu
The paper introduces an explainable pedestrian routing method that estimates perceived safety by first generating a natural‑language caption from street‑level images and then deriving risk scores solely from structured features of that caption. Benchmarking nine captioning setups against a CLIP image‑embedding baseline shows comparable performance, and the system was deployed on over 650,000 images across 36 wards in Manchester and Huddersfield. Field validation with 3,669 ratings from 70 participants revealed a modest but statistically significant correlation (r = 0.262) with human judgments, while a stronger supervised benchmark did not translate into better real‑world performance.
By Simon Parkinson, Paloma Liu, Wei Zheng, Mohammadreza Sheikhfathollahi
Vision-Language Models (VLMs) such as CLIP are now foundational to multimodal systems, yet their robustness to spurious correlations remains poorly understood at scale. We present the first large-scale empirical study of 194 publicly available VLMs, including 16 model families, covering a wide range of model sizes, 24 training datasets, and three evaluation benchmarks, namely ImageNet (overall performance), CelebA (typical single-attribute bias), and UrbanCars (complex multi-attribute biases).
BiasGym is a cost‑effective, generalizable framework that injects specific biases into large language models via token‑based fine‑tuning while keeping the model frozen. It then uses two debiasing methods—Scope and Steer—to identify and suppress or redirect the components responsible for biased behavior. The framework enables consistent bias elicitation, precise localization of bias associations, and targeted debiasing without harming downstream performance, and it has been shown to reduce real‑world stereotypes such as labeling Italians as reckless drivers.
By Sekh Mainul Islam, Nadav Borenstein, Siddhesh Milind Pawar, Haeun Yu, Arnav Arora, Isabelle Augenstein