arXiv AI
NewsBench: Expert-Grounded Evaluation of Epistemic Quality in AI News Reporting
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
3d ago
Distilling Expert Judgment at Scale
The paper proposes a method to capture expert human judgment and reasoning for automated evaluation systems that can scale across high‑stakes domains such as healthcare, defense, finance, governance, and politics. By assembling expert networks and encoding their reasoning into structured editorial standards, the authors demonstrate that automated judges trained on these standards outperform uncalibrated frontier models on source‑quality and neutrality metrics in AI news reporting. The study validates the approach with strong inter‑annotator agreement and high F1 scores, suggesting a viable framework for expert‑calibrated evaluation in any domain where expertise is scarce.
arXiv Computation and Language
Sep 22
From Articles to Publishers: Aggregating Language Model Predictions for News Source Reliability Inference
arXiv:2609.24219v1 Announce Type: new Abstract: Traditionally, the reliability of news publishers is assessed by expert organisations that evaluate editorial practices, transparency and factual stand...
arXiv AI
Sep 1
An Explainable Coherence Score for Detecting Temporal Inconsistencies in Political News
The paper presents the Temporal Coherence Score (TCS), a continuous, interpretable metric for detecting temporal inconsistencies in political news. TCS is computed through a four‑stage pipeline that extracts temporal facts, builds a temporal knowledge graph, verifies consistency using internal rules and external references, and aggregates scores with explanations. On a benchmark of 100 political articles with injected errors, TCS achieves 0.909 precision, providing detailed explanations for each flagged inconsistency.
arXiv AI
Aug 25
LitReview Arena: Evaluating Literature Review Agents with Battle-Style Peer Review Platform
arXiv:2608.21374v1 Announce Type: new Abstract: Literature reviews are essential to scientific progress, but rigorously evaluating automatically generated reviews remains difficult because many aspec...
arXiv Computation and Language
Sep 15
Can We Trust the Judges? Validation of Factuality Evaluation Methods via Answer Perturbation
arXiv:2609.15561v1 Announce Type: new Abstract: Evaluating the factual correctness of large language models (LLMs) is vital for many applications. But are our evaluation tools themselves trustworthy?...
arXiv Computation and Language
Oct 1
A Missing Piece for Trustworthy AI Reviewers: From Benchmarking Rhetorical Robustness to SciCore Review
arXiv:2609.39027v1 Announce Type: new Abstract: AI reviewers can assign different judgments to manuscripts that report the same science in different wording, potentially rewarding rhetorical optimiza...