arXiv:2603. 03989v2 Announce Type: replace-cross Abstract: When visual evidence is ambiguous, vision models must decide how to interpret face-like patterns.
The study compares human and vision‑language model (VLM) responses to cross‑modal association tasks, using identical stimuli (a pseudo‑word and two images) and recording both choices and eye movements. While larger VLMs show some alignment with human choices, their attention patterns correlate poorly with human gaze, performing no better than a simple center‑bias baseline. Fine‑tuning VLMs on human choices improves choice alignment but not attention alignment, and training on human gaze improves attention correlation without affecting choice accuracy.
arXiv:2610.35017v1 Announce Type: new Abstract: Text-steerable visual encoders condition a frozen Vision Transformer on a natural-language prompt and point at the matching image region, producing pro...
CounterCount is a diagnostic dataset that pairs factual and counterfactual images to test whether Vision‑Language Models (VLMs) rely on visual evidence or on world knowledge when counting attributes. Humans achieve near‑perfect accuracy on both types of images, whereas VLMs drop from 91.6% to 49.5% on counterfactuals, often defaulting to canonical counts. The study shows that VLMs exhibit a System‑1‑like bias, underutilize visual evidence, and that targeted input manipulations can mitigate this bias.
Text-steerable visual encoders condition a frozen Vision Transformer on a natural-language prompt and point at the matching image region, producing prompt-conditioned heatmaps and features. We audit t...
EXPL-FR is a lightweight adapter that aligns a vision‑language model’s image encoder with a frozen face‑recognition (FR) embedding space, enabling the FR model to be explained using semantic attribute prompts without any text training. By mapping 978 attribute prompts across 22 categories into the FR space, the method identifies the most detectable concepts—forming a readable semantic signature that better separates identities than the full vocabulary. The approach is evaluated on four FR backbones and two VLM encoders, providing identity‑level, per‑image, and differential explanations, and demonstrates that prompt‑driven audits can rank FR models by per‑ethnicity error and attribute‑change verification cost without requiring labeled data.
arXiv:2608. 14130v1 Announce Type: cross Abstract: Computer vision models for generated facial content, such as face editing and privacy protection, increasingly affect people, requiring similarity metrics that serve as faithful proxies for human perception.
The paper investigates whether adding textual personas to vision‑language models (VLMs) causes the models to perceive images as the described individuals do. Experiments across 14 VLMs show that persona prompts never outperform simple rule‑based baselines and actually erase real differences between groups such as people with low vision, color blindness, or hearing loss. While the models can predict how visual impairments affect humans, the personas do not alter the models’ image representations or answer probabilities, indicating that the models merely state what they know about people rather than seeing like them.
The paper introduces EmoNet‑Face‑HQ, a benchmark of 2,500 AI‑generated portraits each evaluated by four psychology experts across 40 emotion categories. It shows that most vision‑language models (VLMs) perform no better than random guessing unless fine‑tuned, but that changing the prompt to a simple yes/no per category can raise a VLM’s agreement with experts from chance to near expert levels. When evaluated using probability‑of‑yes scores on the five most‑agreed‑upon categories, eleven open‑weight VLMs achieve a quadratic‑weighted κ of 0.425, below the 0.468 two‑expert agreement, yet their average probability‑of‑yes score (0.553) matches the benchmark’s fine‑tuned model and outperforms it in three cases without any training.
The study audits six vision‑language models (VLMs) to assess whether they consistently encode affective qualities of 3D shapes, using Kansei adjective pairs as affective axes. Across ten ShapeNet categories, models show moderate agreement (mean rank correlation 0.36) that is lower than geometric controls but higher than unrelated adjective pairs, with convergence varying widely by category and axis. The authors demonstrate how this audit informs a UI prototype that selectively exposes Kansei descriptors for generative design interfaces.
arXiv:2608.30964v1 Announce Type: new Abstract: Pretrained vision embeddings are increasingly used as general-purpose representations for modelling how people appraise urban scenes, and are validated...
The paper argues that many vision‑language model (VLM) hallucination metrics mistakenly treat every object mentioned in a caption as a claim that the object is present. It introduces NAME‑Bench, a dataset of 25,497 non‑assertive mentions labeled as depictive, purpose, negation, or hedged, and shows that current detectors flag these as hallucinations at high rates. Adding non‑assertive mentions as negatives improves detection on LLaVA‑1.5 but requires much more data for newer VLMs, indicating that conflating mention with assertion is a persistent problem.