AesCanvas is a new dataset and benchmark that evaluates image aesthetic models on two fronts: CritiqueCanvas, which contains 519,136 instruction–response pairs for long‑form, multi‑dimensional critique across photography, painting, and virtual imagery, and ContextCanvas, which offers 301 expert‑reviewed use scenarios to assess contextual aesthetic suitability. The benchmark tests closed‑source, open‑weight general, and aesthetic‑specific multimodal large language models, revealing that models excel at critique generation but lag in context‑sensitive judgment. The study shows that aesthetic specialization does not reliably transfer to contextual suitability and highlights the need for culturally situated, evidence‑grounded suitability as a distinct objective for aesthetic modeling.
By Xuanwei Hu, Haoyu Dong, Kejun Wu, Tianyi Liu, Jianjun Gao
The study examined whether reasoning-enhanced large language models produce psychologically deeper short stories. Using 60 blinded pairwise evaluations, human annotators found no consistent advantage for reasoning-oriented models: GPT‑5 slightly outperformed GPT‑4o, while DeepSeek‑R1 lagged behind DeepSeek‑V3. In contrast, an LLM-as-a-Judge consistently favored reasoning models, revealing a strong pro‑reasoning bias that may mask reader subjectivity in creative tasks.
By Ruichen Zheng, Yihe Wang, Fabrice Harel-Canada, Sara Khosravi, Zeynep Senahan Yildiz, Amit Sahai, Nanyun Peng
Melange is a synthetic visual bias benchmark designed to evaluate social biases in multi‑modal large language models (MLLMs). It extends the language‑only BBQ dataset by generating controlled visual scenarios that clearly depict target attributes, and validates images through automated VQA screening and human inspection. Using Melange, 23 open‑source MLLMs were tested in text‑only, image‑only, and image‑with‑context settings, revealing that switching from text to vision reduces fairness by an average of 0.291 and that models often choose stereotypical responses when biased answers are made.
By Razieh Rezaei, Ehsaneddin Asgary, Masoud Jalili Sabet
The paper investigates how large language models (LLMs) express subjective preferences in short-form content such as numbers, baby names, and fictional cities. It identifies 29 aspects (e.g., length, alphabetic order, popularity) that influence model choices and shows that these preferences are strong, reproducible, and sensitive to how questions are phrased. The study also reveals that the interaction between generating alternatives and ranking them can produce unexpected preference patterns, raising concerns about potential manipulation.
By Katy Blumer, Jon Kleinberg, Ravi Kumar, Andrew Tomkins
The paper introduces EmoNet‑Face‑HQ, a fine‑grained emotion recognition benchmark that uses generated portraits and a 40‑category taxonomy to evaluate vision‑language models (VLMs). It finds that VLMs perform poorly when asked to generate responses but can match or surpass a fine‑tuned model (Empathic‑Insight‑Face) when their logits are read directly as binary queries. The study shows that the benchmark’s difficulty lies in the readout process rather than in perception, and that graded probability outputs yield better performance than simple yes/no questions.
By Tobias Hallmen, Fabian Deuser, Robin-Nico Kampa, Norbert Oswald, Elisabeth Andr\'e
arXiv:2606. 02578v1 Announce Type: cross Abstract: Recent multimodal large language models have demonstrated strong reasoning ability, yet their reliability as automated evaluators remains limited by a critical weakness: when visual evidence conflicts with textual cues, MLLM judges tend to reward plausible narratives over perceptually correct answers.
By Seojeong Park, Jiho Choi, Junyong Kang, Seonho Lee, Jaeyo Shin, Hyunjung Shim
arXiv:2609.13773v1 Announce Type: new
Abstract: LLM-as-a-Judge evaluators are increasingly used to score open-ended generation, yet a judge's correlation with human ratings on its development set may...
By Ruichen Zheng, Yihe Wang, Fabrice Y Harel-Canada, Sara Khosravi, Zeynep Senahan Yildiz, Amit Sahai, Nanyun Peng
arXiv:2610.05094v1 Announce Type: cross
Abstract: Researchers increasingly use Large Language Models as judges (LLM-as-a-judge) to evaluate model outputs. Yet there are no standards for how to design...
By Laur\`ene Vaugrante, Thilo Hagendorff
The paper investigates whether Multimodal Large Language Models (MLLMs) can reason about visual persuasion. It finds that current models often shortcut by citing obvious visual elements instead of truly reasoning about diverse cues, and proposes a fine‑tuning method using multi‑perspective rationales along with an evaluation framework that separates prediction performance from rationale faithfulness. The study shows that while fine‑tuning improves prediction, higher accuracy does not guarantee more faithful reasoning, underscoring the need for distinct evaluation of effectiveness and faithfulness.
By Naeun Lee, Hyunjong Kim, Sunghwan Choi, Injin Kong, Yohan Jo
arXiv:2607. 16232v1 Announce Type: cross Abstract: The growing use of statistical learning algorithms to infer human preferences from high-dimensional choice data runs up against a fundamental challenge: choice alternatives typically differ in many ways simultaneously, so it is generally unclear which factors actually drove an observed decision and should be credited as preferences.
By Zachary Wojtowicz, Ayush Nayak, Jacob Andreas
Traditional Image Aesthetic Assessment (IAA) methods mainly rely on regressing absolute Mean Opinion Scores (MOS). However, such a paradigm overlooks the inherently dynamic nature of human aesthetic perception, which relies on subconscious comparison against implicit visual references.
arXiv:2605.28313v2 Announce Type: replace
Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities in tasks related to reasoning and judgment. However, assessing the quality o...
By Nicol\'as Benjam\'in Ocampo, Agnes Paullate Nyiranziza, Davide Ceolin