arXiv AI
The human-authorship halo: attribution bias in literary style evaluation by humans and AI
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Oct 31
Beauty Is Human: When People Love AI-Generated Art, They Think a Human Made It
The study shows that when people enjoy an artwork, they are more likely to believe it was created by a human, regardless of whether it was actually AI-generated or not. In a large online Turing Test, 22.1 percentage points more of respondents who favored a picture thought it was human-made, and this pattern held across all images and detection methods. Even those who were skeptical of AI art often preferred AI-generated pieces but still attributed them to humans, indicating a strong link between aesthetic preference and perceived authorship.
arXiv AI
Aug 26
The Limits of Automatic Evaluation of Creativity in Large Language Models
The paper examines whether existing automatic methods can reliably assess creativity in text produced by large language models (LLMs). By collecting human ratings on 11 creativity dimensions for both human and AI short stories, the authors compare these judgments with automated metrics and LLM-as-a-Judge evaluations. The results show a significant misalignment: automated metrics and LLM judges favor AI-generated stories and show near-zero correlation with human assessments, revealing fundamental limitations in current computational approaches to evaluating creative text.
arXiv AI
Aug 20
Self- and Other-Labels Induce Bidirectional Bias in LLM Judges
The study investigates bias in large language model (LLM) judges by having ten LLMs evaluate narrative constraint selections rather than generated text. Results show that self-preference largely disappears under blind evaluation when quality and evaluator severity are controlled, but self- and other-labels alone shift scores bidirectionally when quality is matched. The authors conclude that authorship attribution drives evaluation bias and that open-ended, ground‑truth‑free tasks can effectively study LLM judge behavior.
Hugging Face Trending Papers
Aug 3
Style Wins, Substance Loses: A Diagnosis of LLM-as-Judge in Idea Generation
However, whether these judges truly evaluate the scientific substance of ideas or are influenced by superficial stylistic presentation remains an open question. To address this question, we propose SciStyleBench, a unified three-component benchmark for diagnosing and mitigating stylistic bias in LLM-based idea evaluation: (i) First, SciStyleStage, a three-stage evaluation environment that applies controlled stylistic perturbations to fixed scientific content across three settings no context, fixed-domain context, and open-domain retrieval context, covering 600 scientific ideas and 15 style variants, with 9,000 evaluation instances per setting; (ii) Second, SciStyleMetrics, a set of quantitative measures, including Style Bias Index (SBI), Substance Recognition Rate (SRR), and Adversarial Win Rate (AWR), to characterize how stylistic variation affects scoring stability, substance discrimination, and ranking robustness; (iii) Third, SciStyleExtractor, a plug-and-play evaluation module that separates presentation style from scientific content by predicting style type and deviation before style-conditioned evaluation, enabling us to assess whether style awareness reduces stylistic bias.
arXiv AI
Aug 5
Style Wins, Substance Loses: A Diagnosis of LLM-as-Judge in Idea Generation
arXiv:2608. 01666v2 Announce Type: replace-cross Abstract: However, whether these judges truly evaluate the scientific substance of ideas or are influenced by superficial stylistic presentation remains an open question.
arXiv AI
Sep 15
Authorship attribution and aesthetic evaluation of AI poetry: a case study with Haiku
arXiv:2609.15511v1 Announce Type: cross Abstract: This paper investigates the generation and human evaluation of Japanese haiku by contemporary Large Language Models (LLMs), focusing on authorship pe...