arXiv Computation and Language
Human-Anchored Factuality Evaluation with Strategic Annotation
Read the original on arXiv Computation and Language →The Flow has not summarised this story yet — read it at arXiv Computation and Language.
Related stories
arXiv AI
Jun 2
Who Annotates in NLP? A Large-scale Assessment of Human Annotation Reporting between 2018 and 2025
arXiv:2606. 02255v1 Announce Type: cross Abstract: Human annotation is the empirical foundation of much NLP research, from dataset construction to model evaluation, but papers often leave unclear who produced the annotations and how the annotation process was controlled.
arXiv AI
Jun 9
Margin-Adaptive Confidence Ranking for Reliable LLM Judgement
arXiv:2605. 15416v2 Announce Type: replace-cross Abstract: Jung et al.
arXiv Machine Learning
Jul 21
BACON: Budgeted Human Calibration for Modeling and Evaluation with Multiple AI Judges
arXiv:2607. 16239v1 Announce Type: new Abstract: AI judges offer a scalable, low-cost alternative to human evaluation, but their outputs can be biased relative to human preferences and highly item-dependent, varying across judges, tasks, and domains.
arXiv AI
Jun 3
Fixing FOLIO and MALLS: Verified Annotations and an LLM-assisted Framework to Focus Human Relabeling
arXiv:2606. 02837v1 Announce Type: cross Abstract: Accurate translation from Natural Language to First-Order Logic (NL-to-FOL) underpins neurosymbolic AI systems and Natural Language Inference (NLI), making the quality of NL-to-FOL benchmarks essential -- yet these datasets have never been rigorously audited.
arXiv AI
Jun 16
Metric Match: A Subset Selection Approach to Evaluating LLM Judge Reliability
arXiv:2606. 15029v1 Announce Type: new Abstract: LLM judges are used to reduce the need for costly human labor in evaluating open-ended text generation.
arXiv Machine Learning
Jun 2
How to Correctly Report LLM-as-a-Judge Evaluations
arXiv:2511. 21140v4 Announce Type: replace Abstract: Large language models (LLMs) are widely used as scalable evaluators of model responses in lieu of human annotators.