The paper proposes a method to capture expert human judgment and reasoning for automated evaluation systems that can scale across high‑stakes domains such as healthcare, defense, finance, governance, and politics. By assembling expert networks and encoding their reasoning into structured editorial standards, the authors demonstrate that automated judges trained on these standards outperform uncalibrated frontier models on source‑quality and neutrality metrics in AI news reporting. The study validates the approach with strong inter‑annotator agreement and high F1 scores, suggesting a viable framework for expert‑calibrated evaluation in any domain where expertise is scarce.
By Robbie Goldfarb, Andrew B. Hall, Jillian Fisher, Kathryn Salam, Matt Wilde
arXiv:2609.24219v1 Announce Type: new
Abstract: Traditionally, the reliability of news publishers is assessed by expert organisations that evaluate editorial practices, transparency and factual stand...
By John Bianchi, Manuel Pratelli, Fabio Pinelli, Marinella Petrocchi
The paper presents the Temporal Coherence Score (TCS), a continuous, interpretable metric for detecting temporal inconsistencies in political news. TCS is computed through a four‑stage pipeline that extracts temporal facts, builds a temporal knowledge graph, verifies consistency using internal rules and external references, and aggregates scores with explanations. On a benchmark of 100 political articles with injected errors, TCS achieves 0.909 precision, providing detailed explanations for each flagged inconsistency.
By Marius Nicusor Pantea, Adrian Groza
arXiv:2608.21374v1 Announce Type: new
Abstract: Literature reviews are essential to scientific progress, but rigorously evaluating automatically generated reviews remains difficult because many aspec...
By Ruotong Zhao, Zhiyu Chen, Xurui Liu, Haidong Xue, Dong Liang, Jigao Fu, Wu YanBiao, Yuanyi Zhen, Fengli Xu, Yong Li
arXiv:2609.15561v1 Announce Type: new
Abstract: Evaluating the factual correctness of large language models (LLMs) is vital for many applications. But are our evaluation tools themselves trustworthy?...
By Sarra Gharsallah, Adele Robaldo, Mariia Tokareva, Giovanni Gatti Pinheiro, Ilyana Guendouz, Rapha\"el Troncy, Paolo Papotti, Pietro Michiardi
arXiv:2609.39027v1 Announce Type: new
Abstract: AI reviewers can assign different judgments to manuscripts that report the same science in different wording, potentially rewarding rhetorical optimiza...
By Chenguang Wang, Ming Li, Chengrui Fan, Jianpeng Chen, Han Chen, Tianyi Zhou, Dawei Zhou
MABPD (Multi‑Agent Bias Probing & Detection) is a training‑free pipeline that uses three specialized large language model agents to analyze news articles from complementary perspectives and resolve disagreements via a Structured Argument Debate (SAD) protocol. SAD imposes an asymmetric burden of proof—biased claims lacking grounded textual evidence receive zero weight—along with role‑weighted voting and post‑consensus verification, replacing task‑specific supervised decision boundaries. Ablation studies show that the debate module alone accounts for up to a 10.6‑point F1 gain, and on the BABE benchmark MABPD attains 83.4% macro F1, within 0.7 percentage points of the supervised state‑of‑the‑art, while achieving 75.0% zero‑shot accuracy on the SemEval 2019 HyperPartisan corpus.
By Garvit Joshi (Graphic Era University, Dehradun, India), Stavya Dhyani (Graphic Era University, Dehradun, India), Jasmine (Graphic Era University, Dehradun, India), Arun Chauhan (Graphic Era University, Dehradun, India)
arXiv:2609.27165v1 Announce Type: cross
Abstract: Large language models (LLMs) are increasingly used to measure public value orientations from long social media posts, yet such posts often mix backgr...
By Yuhe Wu, Rui Qian, Guangyu Wang, Yuran Chen, Yuanchao Zhu, Junjie Yang, Zhengheng Li, Jiulin Cai, Tianyi Zhang, Zihan Dong, Jiaxin Liu, Yujie Chen, Guang Zhang
arXiv:2604. 12138v2 Announce Type: replace Abstract: This position paper argues that Retrieval-Augmented Generation systems exhibit a systematic factual bias-optimizing for epistemic uncertainty reduction while ignoring the aleatoric uncertainty inherent in opinion-rich content - and that this misalignment demands a paradigm shift in retrieval system design.
By Aditya Agrawal, Alwarappan Nakkiran, Darshan Fofadiya, Alex Karlsson, Harsha Aduri
arXiv:2607. 18983v1 Announce Type: cross Abstract: We present AutoJourn, a demonstration system for multi-perspective news generation and bias-aware evaluation using large language models (LLMs).
By Himel Ghosh, Ahmed Mosharafa, Georg Groh
arXiv:2609.00494v1 Announce Type: new
Abstract: LLM-based factuality judges provide scalable evaluation signals, but their metrics are often systematically biased relative to human judgments. We stud...
By Yu Wang, Craig Erickson, Kevin Small
Large language models are increasingly used as automated reviewers in scientific evaluation, creating a recursive feedback loop where later reviewers learn from earlier model-generated judgments. A study using Llama 3.1 8B fine‑tuned on ICLR reviews shows that incorporating synthetic reviews compresses rating distributions and reduces semantic diversity, a phenomenon termed scientific‑judgment collapse. To counter this, the authors introduce TrustReviewer, an open‑source LLM system that curates training data and applies paired activation steering at test time to preserve judgment diversity and improve recommendation alignment.
By Sy-Tuyen Ho, Minghui Liu, Furong Huang