arXiv Computation and Language September 23, 2026 By Mizanur Rahman, Huan Wu, Arash Asgari, Enamul Hoque Prince, Laleh Seyyed-Kalantari

Same Chart, Different Story: Bias in Vision-Language Chart Interpretation

Read the original on arXiv Computation and Language →

The paper introduces ChartBias, a benchmark of 820 real-world charts covering six social attributes, designed to audit bias in vision‑language models (VLMs) that interpret charts. Across 12 VLMs, the study identifies three failure modes—narrative shift, group hallucination, and preference polarity—where models produce different or misleading narratives when the referenced social group changes. A multi‑agent mitigation framework is proposed, separating evidence extraction from group‑conditioned generation and using a counterfactual judge, which reduces narrative shift while maintaining chart‑grounded reasoning.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

arXiv AI

Jul 28

Chart Deception in Vision-Language Models: From Vulnerability to Mitigation

arXiv:2607. 22600v1 Announce Type: new Abstract: Information visualizations are widely used to communicate patterns, trends, and outliers, yet deceptive design choices-such as truncated or inverted axes, distorted aspect ratios, inappropriate encodings, and misleading color mappings-can systematically alter interpretation while preserving the underlying data.

By Ridwan Mahbub, Mohammed Saidul Islam, Md Tahmid Rahman Laskar, Mizanur Rahman, Mir Tafseer Nayeem, Enamul Hoque

llms agents multimodal benchmarks

arXiv AI

Aug 25

Mitigating Bias in Large Vision-Language Models via Counterfactual Ensemble Decoding

arXiv:2608.21415v1 Announce Type: cross Abstract: Large Vision-Language Models (LVLMs) have achieved remarkable performance across a wide range of tasks; however, they often inherit social biases fro...

By Yisong Xiao, Aishan Liu, Yongxin Huang, Zonghao Ying, Shiji Zhao, Tianlin Li, Yong Han, Jian Yang, Xianglong Liu

llms multimodal benchmarks safety

arXiv Machine Learning

Sep 1

Tracing the Latent Threads: A Mechanistic Study of How LLMs Represent and Operationalize Race and Ethnicity Cues

arXiv:2601.12868v2 Announce Type: replace-cross Abstract: Large language models (LLMs) increasingly operate in high-stakes settings where demographic attributes such as race and ethnicity may be expl...

By Shiyue Hu, Ruizhe Li, Yanjun Gao

llms nlp safety

arXiv AI

Jul 15

Navigating the Mirage: A Dual-Path Agentic Framework for Robust Misleading Chart Question Answering

arXiv:2603. 28583v2 Announce Type: replace-cross Abstract: Despite the success of Vision-Language Models (VLMs), misleading charts remain a significant challenge due to their deceptive visual structures and distorted data representations.

By Yanjie Zhang, Yafei Li, Rui Sheng, Zixin Chen, Yanna Lin, Huamin Qu, Lei Chen, Yushi Sun

llms agents computer-vision nlp fine-tuning efficiency multimodal benchmarks safety

arXiv Computation and Language

Sep 2

Evaluating Second-Order Bias of LLMs Through Epistemic Entitlement

The paper introduces a novel framework for assessing second‑order bias in large language models (LLMs), defined as bias in how an LLM judges the acceptability of biased content. Using principles from entitlement epistemology, the authors design a reasoning task that asks LLMs to determine whether a biased text is acceptable for specific demographic groups, and propose two metrics to quantify biased judgments. Experiments on both open‑source and closed‑source models reveal that the task bypasses safety guardrails, uncovers systematic variations across target groups, and demonstrates that models still rely on demographic labels when evaluating bias.

By Ramaravind Kommiya Mothilal, Terry Jingchen Zhang, Raiyan Ahmed, Zhijing Jin, Shion Guha, Syed Ishtiaque Ahmed

llms agents nlp safety

arXiv Computer Vision

Sep 1

Guardrail-Agnostic Societal Bias Evaluation in Large Vision-Language Models

arXiv:2608.29590v1 Announce Type: new Abstract: We propose a societal bias evaluation method for large vision-language models (LVLMs) in the era of strong safety guardrails. Existing benchmarks rely...

By Yusuke Hirota, Michael Ross Boone, Arun George Zachariah, Jibin Rajan Varghese, Yu-Chiang Frank Wang, Boyi Li, Ryo Hachiuma

llms multimodal benchmarks safety

One email a morning, machine-written

Related stories

Chart Deception in Vision-Language Models: From Vulnerability to Mitigation

Mitigating Bias in Large Vision-Language Models via Counterfactual Ensemble Decoding

Tracing the Latent Threads: A Mechanistic Study of How LLMs Represent and Operationalize Race and Ethnicity Cues

Navigating the Mirage: A Dual-Path Agentic Framework for Robust Misleading Chart Question Answering

Evaluating Second-Order Bias of LLMs Through Epistemic Entitlement

Guardrail-Agnostic Societal Bias Evaluation in Large Vision-Language Models