arXiv Computation and Language November 4, 2026 By Tobias Hallmen, Fabian Deuser, Robin-Nico Kampa, Norbert Oswald, Elisabeth Andr\'e

Parsed Text Understates Vision-Language Models: From Below to Above Expert Agreement on Fine-Grained Facial Emotion

Read the original on arXiv Computation and Language →

The paper introduces EmoNet‑Face‑HQ, a benchmark of 2,500 AI‑generated portraits each evaluated by four psychology experts across 40 emotion categories. It shows that most vision‑language models (VLMs) perform no better than random guessing unless fine‑tuned, but that changing the prompt to a simple yes/no per category can raise a VLM’s agreement with experts from chance to near expert levels. When evaluated using probability‑of‑yes scores on the five most‑agreed‑upon categories, eleven open‑weight VLMs achieve a quadratic‑weighted κ of 0.425, below the 0.468 two‑expert agreement, yet their average probability‑of‑yes score (0.553) matches the benchmark’s fine‑tuned model and outperforms it in three cases without any training.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

arXiv Computation and Language

Oct 7

The Failure Is in the Readout: Fine-Grained Emotion Recognition Benchmarks Measure Elicitation, Not Perception

The paper introduces EmoNet‑Face‑HQ, a fine‑grained emotion recognition benchmark that uses generated portraits and a 40‑category taxonomy to evaluate vision‑language models (VLMs). It finds that VLMs perform poorly when asked to generate responses but can match or surpass a fine‑tuned model (Empathic‑Insight‑Face) when their logits are read directly as binary queries. The study shows that the benchmark’s difficulty lies in the readout process rather than in perception, and that graded probability outputs yield better performance than simple yes/no questions.

By Tobias Hallmen, Fabian Deuser, Robin-Nico Kampa, Norbert Oswald, Elisabeth Andr\'e

llms robotics fine-tuning multimodal benchmarks safety

arXiv AI

Sep 28

From Reward Signal to Visual Utility: A Controlled Audit of Medical VLM Post-Training

arXiv:2609. 31450v1 Announce Type: cross Abstract: Medical vision-language model (VLM) post-training is commonly evaluated through answer accuracy.

By Wang Jingxin

llms fine-tuning multimodal

arXiv Computation and Language

1d ago

Same Outputs, Different Scores: Auditing the Unchecked Graders of Multimodal LLM Benchmarks

The paper investigates how the scoring rules used in multimodal large‑language‑model (LLM) benchmarks can dramatically alter reported accuracy. By reproducing official scorers for eight benchmarks and comparing them to a blinded human‑plus‑LLM adjudication, the authors show that the same model outputs can yield accuracy differences of up to 22 points depending on the rule. They argue that no single scoring rule is reliable and recommend that benchmarks report non‑compliance rates and use at least two distinct scoring rules.

By Ahmad Jarjis Hasan, Jonathan Lwowski

llms multimodal benchmarks

arXiv Computer Vision

Oct 12

Auditing VLM Benchmarks from Existing Evaluation Runs

The paper introduces an audit layer for vision‑language model benchmarks that reuses per‑question artifacts from evaluation runs to assess the benchmark itself. Six detectors analyze saved outputs, and an additional text‑only rerun removes visual inputs to produce per‑question reports and aggregate statistics, including permutation‑null excess and interval estimates. Applying this method to the MMStar benchmark reveals that 20.2% of its questions show no accuracy gain from images, with a small but significant excess in mathematics and science categories, and identifies questions where all models chose the same non‑gold option.

By Denis Degtiarev, Anna Gromova, Sergey Katkov, Artem Mikhailin, Safina Mukhamedshina, Albert Nasybullin, Karina Shaikhutdinova, Artem Tuzov

llms multimodal benchmarks

arXiv Computer Vision

Sep 1

Frontier vision-language models have overtaken young adults at detecting AI-generated portraits -- but not their calibration

arXiv:2608.30210v1 Announce Type: cross Abstract: AI image generators now create face portraits that are hard to tell from real photographs. Vision-language models (VLMs) are increasingly proposed to...

By Sunwhi Kim (Hwasung Medi-Science University, Dept. of Bio-Healthcare), Sunyul Kim (Yonsei University, Graduate School of Engineering, Dept. of Artificial Intelligence), Meounggun Jo (Hoseo University), Jini Tae (Gwangju Institute of Science and Technology, School of Humanities and Social Sciences)

llms multimodal

arXiv AI

Oct 21

CounterCount: Diagnosing Knowledge Bias in How Vision-Language Models Count Attributes

CounterCount is a diagnostic dataset that pairs factual and counterfactual images to test whether Vision‑Language Models (VLMs) rely on visual evidence or on world knowledge when counting attributes. Humans achieve near‑perfect accuracy on both types of images, whereas VLMs drop from 91.6% to 49.5% on counterfactuals, often defaulting to canonical counts. The study shows that VLMs exhibit a System‑1‑like bias, underutilize visual evidence, and that targeted input manipulations can mitigate this bias.

By Reem Alzahrani, Bushra Bin Hemid, Hassan Alshanqiti, Zaid Alyafeai, Paul Gavrikov, Bernard Ghanem, Abdelrahman Eldesokey

llms multimodal safety