The paper introduces a modular data‑science pipeline that estimates public sentiment toward individuals using fragmented, unstructured open‑source intelligence. The pipeline combines web search, text extraction, relevance filtering, tokenisation, co‑reference resolution, and sentiment analysis to produce auditable person‑level sentiment distributions. By comparing AFINN, VADER, and the domain‑specific MINOS algorithm, the authors show that MINOS best distinguishes positive, ambiguous, and negative reputational cases, and they apply the method to the UK Honours system to support transparent, reproducible, human‑in‑the‑loop sentiment assessment for high‑stakes decisions.
arXiv:2607. 09324v1 Announce Type: cross Abstract: Identifying and assigning keywords at scale is a technical, practical, and ethical challenge for crowdsourced collections.
arXiv:2603. 20450v2 Announce Type: replace-cross Abstract: A number of scientific conferences and journals have recently enacted policies that prohibit LLM usage by peer reviewers, except for polishing, paraphrasing, and grammar correction of otherwise human-written reviews.
arXiv:2608.21376v1 Announce Type: cross Abstract: Many NLP tasks require systems to provide attribution in their outputs--i.e. citations to grounding sources. Attribution serves as a bulwark against...
arXiv:2607. 28282v1 Announce Type: cross Abstract: Evaluating the quality and relevance of textual outputs from Large Language Models (LLMs) remains challenging and resource-intensive.
The paper introduces SAGE, a lightweight linguistic framework that detects AI-generated peer reviews by exploiting subjectivity at semantic and pragmatic levels. It uses speech act annotations and model-predicted subjectivity scores to train a classifier achieving an AUROC of 0.9566. When combined with the Sem-Detect detector, SAGE boosts the true positive rate at 1% false positive rate on the APRDB benchmark from 0.874 to 0.985, surpassing current state‑of‑the‑art methods.
arXiv:2601. 21817v2 Announce Type: replace-cross Abstract: Evaluating large language models (LLMs) on open-ended tasks without ground-truth labels is increasingly done via the LLM-as-a-judge paradigm.
arXiv:2610.33240v1 Announce Type: new Abstract: As frontier AI systems increasingly serve as sources of political and current-events information, evaluating the epistemic quality of their responses i...
The paper introduces WSF-ARG+, a new dataset that pairs hate speech with check‑worthiness annotations, and presents an LLM‑in‑the‑loop framework to streamline the annotation process. Experiments with 12 open‑weight large language models demonstrate that the framework cuts human effort while maintaining annotation quality. The study also shows that incorporating check‑worthiness labels improves hate‑speech detection performance, boosting macro‑F1 scores for large models by up to 0.213 and averaging 0.154 across models.
Understanding moral values in social media text offers insight into moral judgement formation, and supervised NLP models trained on crowdsourced data have achieved strong classification performance. However, most approaches simplify the problem by aggregating multiple annotators' labels into a single "ground truth", overlooking the inherent subjectivity of the task.
arXiv:2606. 05308v1 Announce Type: new Abstract: With PRECISE, we extended Prediction-Powered Inference to produce bias-corrected estimates of ranking evaluation metrics by combining a small human-labeled set with a large LLM-judged set.
arXiv:2609.00319v1 Announce Type: cross Abstract: Online health information seeking is shifting from keyword search, where users consider a ranked list of links, to conversational systems that compos...
arXiv:2611.06384v1 Announce Type: new Abstract: Typed decision models offer structured choices, scores, and probabilities for low-cost automated evaluation. We present a controlled audit of presentat...