arXiv AI By Jiawei Chen

Telling Is Not Seeing: Textual Personas Rarely Make Vision-Language Models Answer Like the People They Describe

Read the original on arXiv AI →

The paper investigates whether adding textual personas to vision‑language models (VLMs) causes the models to perceive images as the described individuals do. Experiments across 14 VLMs show that persona prompts never outperform simple rule‑based baselines and actually erase real differences between groups such as people with low vision, color blindness, or hearing loss. While the models can predict how visual impairments affect humans, the personas do not alter the models’ image representations or answer probabilities, indicating that the models merely state what they know about people rather than seeing like them.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv Computation and Language
Oct 5

Evaluating VQA in Vision Language Models using Cooperative Principles

The paper evaluates Vision Language Models (VLMs) on Visual Question Answering tasks where questions violate Grice's maxims. By generating question modifiers that add non-essential, ambiguous, or false information, the authors show that VLMs such as ChatGPT, Claude, Gemini, and Llava exhibit reduced performance. They also compare human pragmatic reasoning to VLM reasoning, noting differences in how each handles human‑induced versus AI‑generated violations, and find that humans spend less time resolving VLM‑induced violations while VLMs are less accurate in those cases.

By Monika Shah, Sudarshan Balaji, Somdeb Sarkhel, Sanorita Dey, Deepak Venugopal
arXiv AI
Oct 28

A Heart Means Recursion: Exploring Visual Meaning and Communication in VLMs

The paper investigates whether vision‑language models (VLMs) can communicate visually in ways similar to humans. It reports that simple visual symbols, such as a red heart, can unexpectedly steer VLMs toward recursive solutions in unrelated tasks, revealing a generation‑interpretation gap where models produce intuitive shapes that do not effectively influence other models. In more complex communication games, VLMs can creatively use visuals to convey abstract concepts, though some messages remain unintuitive to humans, highlighting both the potential and the challenges of visual communication between humans and VLMs.

By Alexa R. Tartaglini, Zhengxuan Wu, Thadd\"aus Wiedemer, Martin Wattenberg, Been Kim, Yuxuan Li
arXiv Computer Vision
Sep 23

Reading Right, Answering Wrong: How Visual Configuration Changes Affect Evidence Use in VLMs

Vision‑language models (VLMs) can lose accuracy when images are resized, even with minimal changes. The study shows that such small visual configuration changes—like tiling or token arrangement—cause more correctness flips across multiple checkpoints and benchmarks. Interestingly, in many cases the models still read the correct answer but fail to use it, and attention interventions reveal that configuration shifts weaken the use of readable information. By guiding models with field cues and their own transcriptions, the authors correct 97.2% of these errors.

By Dingyang Lin, Yingfeng Luo, Chenglong Wang, Chenwei Zhu, Anxiang Ma, Jingbo Zhu, Tong Xiao