arXiv Computer Vision October 19, 2026 By Yue Dai, Ziyang Liu, Marc Cheong, Caren Han

IMPLICIT-Bench: Stereotyped Defaults and Prompt Following in Text-to-Image Generation

Read the original on arXiv Computer Vision →

IMPLICIT-Bench is a benchmark of 1,831 prompt triplets that tests whether text‑to‑image models can override stereotypical defaults when given explicit requests. Each triplet contains a neutral prompt, a stereotype prompt, and an anti‑stereotype prompt, allowing measurement of a model’s default bias and its ability to follow contradictory instructions. Experiments on Qwen‑Image and Stable Diffusion 3 show that models often default to stereotypes, succeed less often on anti‑stereotype requests, and that prompt rewriting or embedding steering can modestly reduce bias when the relevant stereotype is known.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computer Vision.

arXiv Computer Vision

Sep 22

IMPLICIT-Bench: Measuring Implicit Bias in Text-to-Image Models under Neutral Prompts

arXiv:2609.24228v1 Announce Type: new Abstract: Text-to-image (T2I) models are typically evaluated for bias using slot-based templates such as ``a photo of a [profession]''. Such templates probe only...

By Yue Dai, Ziyang Liu, Marc Cheong, Caren Han

diffusion benchmarks safety

arXiv AI

Oct 5

NegT2IBench: When Negation Changes the Picture. A Polarity Benchmark for Text-to-Image Models

arXiv:2610.03084v1 Announce Type: cross Abstract: Text-to-image (T2I) models are judged by benchmarks that measure whether requested content appears, but these benchmarks largely overlook the complem...

By Omar Elfatairy, Maria A. Bravo, Jessica Bader, Zeynep Akata

diffusion multimodal benchmarks

arXiv Machine Learning

Jun 9

Exposing Hidden Biases in Text-to-Image Models via Automated Prompt Search

arXiv:2512. 08724v3 Announce Type: replace Abstract: Text-to-image (TTI) diffusion models have achieved remarkable visual quality, yet they have been repeatedly shown to exhibit social biases across sensitive attributes such as gender, race and age.

By Manos Plitsis, Giorgos Bouritsas, Vassilis Katsouros, Yannis Panagakis

llms diffusion benchmarks safety

arXiv AI

Jul 22

Attributes Should Come from Images, Not Class Names: Distribution-Conditioned Attribute Selection for Vision-Language Models

arXiv:2607. 18695v1 Announce Type: cross Abstract: A popular route to interpretable zero-shot classification asks a large language model (LLM) to describe each class name and prompts CLIP with the resulting descriptors.

By Gautam Rajendrakumar Gare, Jia Shi, Zhiqiu Lin, Deepak Pathak, John Galeotti, Deva Ramanan

llms rag multimodal

arXiv AI

Sep 17

AMIGO: Agentic Multi-Image Grounding Oracle Benchmark

AMIGO (Agentic Multi-Image Grounding Oracle Benchmark) is a long-horizon evaluation framework for vision‑language models that tests hidden‑target identification across galleries of visually similar images. The benchmark requires a model to ask a sequence of attribute‑focused Yes/No questions, receiving Yes/No/Unsure feedback and penalizing invalid actions with Skip, thereby stressing question selection under uncertainty, constraint tracking, and fine‑grained discrimination. Using the Guess My Preferred Dress task, the study shows that final‑answer accuracy alone overstates performance, as models may guess correctly without verified evidence, waste turns, or violate the protocol, highlighting the need for combined visual discrimination, informative questioning, and robust protocol adherence.

By Min Wang, Ata Mahjoubfar

llms agents multimodal benchmarks

arXiv Computer Vision

Oct 15

Measuring Epistemic Humility in Multimodal Large Language Models

arXiv:2509.09658v3 Announce Type: replace Abstract: Hallucinations in multimodal large language models (MLLMs), where a model generates content inconsistent with the input image, pose significant ris...

By Bingkui Tong, Jiaer Xia, Sifeng Shang, Kaiyang Zhou

llms multimodal benchmarks safety

One email a morning, machine-written

Related stories

IMPLICIT-Bench: Measuring Implicit Bias in Text-to-Image Models under Neutral Prompts

NegT2IBench: When Negation Changes the Picture. A Polarity Benchmark for Text-to-Image Models

Exposing Hidden Biases in Text-to-Image Models via Automated Prompt Search

Attributes Should Come from Images, Not Class Names: Distribution-Conditioned Attribute Selection for Vision-Language Models

AMIGO: Agentic Multi-Image Grounding Oracle Benchmark

Measuring Epistemic Humility in Multimodal Large Language Models