The paper evaluates large language models (LLMs) as data quality annotators on two e-commerce tasks: entity matching and brand mislabeling. In entity matching, a simple rule-based baseline matched the LLM’s zero-shot performance (F1≈0.95), and a few-shot prompt actually lowered performance, highlighting the risk of small-sample prompt tuning. For brand mislabeling, the LLM outperformed a naive rule baseline (F1 0.833 vs 0.721) by leveraging background knowledge, and demonstrated high consistency across repeated runs (99.7% agreement).
By Praphulla Lal Shrestha
arXiv:2608. 10385v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used as relevance assessors in information retrieval (IR) evaluation, raising questions about how assessor framing affects judgment reliability and downstream system comparison.
By Samaneh Mohtadi, Pietro Bernardelle, Joel Mackenzie, Gianluca Demartini
The paper introduces TRAILS, a taxonomy of 38 mechanisms that can compromise the reliability of large language model (LLM) read‑and‑evaluate tasks in evidence synthesis, such as study screening, data extraction, risk‑of‑bias assessment, and critical appraisal. It explains how these mechanisms can cause inconsistent or hidden failures across repeated LLM runs and outlines deployment settings, testing approaches, and mitigation strategies for each. The authors argue that repeatability or accuracy alone does not guarantee trustworthy judgments, emphasizing the need for human evaluation to verify that decisions are evidence‑based, free from external influence, and reproducible.
By Carole Lunny, Ian Goodman, Stefano Berrone, Connor TA Brenna, Emma K Reid, Ivan Perez-Neri, Shailesh Kolekar, Nityanand Jain, Ashwin Kamath, Lucas HCC Santos, Yulia Otmakhova, Heba Hussein, Diana M Cespedes, Gioia Dahdal, Salmaan Kanji
The study audits large language model (LLM) outputs by measuring how well repeated queries recover a collected set of responses versus the full set of possible outputs. Using sample-based rarefaction on 4,500 responses from 50 buying questions across six configurations, the authors find historical-dictionary median recovery rates between 92.6% and 95.2%, which drop to 89.5%–94.7% after re‑adjudicating all candidate strings. Additional analyses with Gemini 3.1 Pro annotations and matched roster data confirm that recovery percentages vary with extraction methods, question selection, and the finite reference collection, underscoring the need for explicit measurement definitions and sensitivity analyses in LLM audits.
By Dmitrij \.Zatuchin
arXiv:2606. 07936v1 Announce Type: cross Abstract: Human evaluation plays a critical role in assessing the quality of generated text.
By Katelyn Xiaoying Mei, Yi-Li Hsu, Minjoon Choi, Zongwan Cao, Chenjun Xu, Bingbing Wen, Su Lin Blodgett, Lucy Lu Wang
The paper presents TRAILS, a taxonomy of 38 mechanisms that can undermine the reliability of large language model (LLM) read‑and‑evaluate tasks in evidence synthesis, such as study screening and risk‑of‑bias assessment. It distinguishes failures that cause inconsistent judgments across repeated runs from those that systematically bias outcomes without obvious signs. The authors outline deployment settings, testing approaches, and mitigation strategies, and argue that repeatability or accuracy alone cannot guarantee trustworthy judgments, emphasizing the need for human evaluation of evidence grounding and process transparency.
The study investigates how prior scores influence large language model (LLM) judgments in the LLM-as-a-Judge paradigm. By testing three prompt conditions—no metadata, revision framing, and anchored metadata containing prior scores—the authors find that prior scores systematically bias evaluations, shifting ratings toward those scores across 192,000 attempts. The bias also affects categorical decisions, blocking 48% of error corrections and flipping 10.18% of correct judgments, and is not mitigated by Chain-of-Thought or a warning, underscoring the need for careful context engineering.
By Ante Kapetanovic, Kemal Altwlkany, Andro Mercep, Tomislav Duricic, Emanuel Lacic
arXiv:2610.03998v2 Announce Type: replace
Abstract: Large language models (LLMs) are increasingly used as synthetic personas representing survey respondents. Their validity as substitutes for particu...
By Khashayar Pourtaheri, Ahmad Zareei
arXiv:2607. 28818v1 Announce Type: new Abstract: As AI companions increasingly mediate repeated social interaction, users may rely on a stable role and shared history, yet locally acceptable replies do not ensure that either persists.
By Pranav Narayanan Venkit, Akshara Prabhakar, Yu Li, Daniel Lee, Chien-Sheng Wu
arXiv:2609.15100v1 Announce Type: cross
Abstract: An image tool can change its underlying generator while retaining its public name, making version attribution from online posts ambiguous. We study t...
By Dennis Ng, Xingyu Shen, Ankit Raj, Kidus Zewde, Tommy Duong, Yuchen Zhou, Yuxin Zhang, Neo Tiangratanakul, Simiao Ren
The paper introduces PAI‑Bench, a benchmark designed to evaluate persistent AI agents on how faithfully they adhere to a versioned identity contract. It separates several dimensions—recall, composition, behavioral enactment, resistance, persistence, lineage, and role‑conditioned updates—while keeping scoring oracles independent of the target process. Experiments on synthetic profiles show that explicit cues can significantly alter the presence of identity identifiers, revealing prompt‑dependent component selection and sensitivity to startup cues.
By Zhenyu Zhao, Roy Zhao
arXiv:2609.23264v1 Announce Type: new
Abstract: Peer-review evaluation is increasingly being automated with LLM-as-a-judge metrics, but this creates a measurement risk. A review may receive a high sc...
By Shakiba Amirshahi, Sajad Ebrahimi, Hai Son Le, Negar Arabzadeh, Ebrahim Bagheri