arXiv:2607. 28282v1 Announce Type: cross Abstract: Evaluating the quality and relevance of textual outputs from Large Language Models (LLMs) remains challenging and resource-intensive.
The paper demonstrates that large language models (LLMs) used for forecasting real‑world events can be manipulated by simply publishing new articles, even without direct access to the model or its retriever. By injecting a small number of targeted news pieces into a common crawl corpus, an adversary can flip over half of the forecast probabilities and significantly degrade forecast accuracy. The study also shows that common defense strategies can be cheaply bypassed, highlighting the vulnerability of probabilistic LLM judgments to information‑supply‑chain attacks.
The paper introduces GPTBIAS, a framework that uses powerful large language models like GPT‑4 to evaluate bias in other LLMs. It employs specially crafted prompts called Bias Attack Instructions to probe for bias and outputs a bias score along with detailed information such as bias types, affected demographics, keywords, reasons, and improvement suggestions. Extensive experiments demonstrate the framework’s effectiveness and usability.
The paper introduces Strategic 16K, a 16,000‑document corpus of diplomatic cables from WikiLeaks’ Public Library of US Diplomacy, designed to eliminate label leakage. It benchmarks six models—both classical machine learning and transformer-based—on this leakage‑controlled dataset, finding BERT and ELECTRA top performers while TF‑IDF with Logistic Regression offers strong accuracy at lower cost. This work provides the first fully reproducible sensitivity‑classification benchmark built under explicit leakage‑control conditions.
The paper presents the Temporal Coherence Score (TCS), a continuous, interpretable metric for detecting temporal inconsistencies in political news. TCS is computed through a four‑stage pipeline that extracts temporal facts, builds a temporal knowledge graph, verifies consistency using internal rules and external references, and aggregates scores with explanations. On a benchmark of 100 political articles with injected errors, TCS achieves 0.909 precision, providing detailed explanations for each flagged inconsistency.
The paper introduces DECO, a diagnostic framework that factorises content into independent moderation criteria, allowing controlled evaluation of large language models (LLMs) at the criterion level. Using pairwise evaluation across four datasets and four LLMs, the authors find that high aggregate benchmark scores can mask significant failures when decisions hinge on specific content aspects required by individual criteria. The study underscores that aggregated labels do not guarantee reliable criterion-conditioned performance, highlighting the need for evaluation methods that explicitly assess this behavior.
arXiv:2606. 04199v1 Announce Type: cross Abstract: The increasing use of large language models has raised concerns about the spread of AI-generated fake news, particularly under varying prompting strategies.
The paper introduces DECO, a diagnostic tool that factorises content into independent criteria for evaluating large language models (LLMs) on content moderation tasks. Using DECO and pairwise evaluation across four datasets and four LLMs, the authors find that high benchmark scores can mask significant failures at the criterion level, especially when decisions hinge on specific content aspects rather than overall harmfulness. The study underscores that aggregated label performance does not guarantee reliable criterion-conditioned evaluation, calling for new methods that explicitly assess this behavior.
arXiv:2609.20838v1 Announce Type: new Abstract: In this study, we examine how modern LLMs generate and detect fake news under controlled settings across four manipulation scenarios. These are open-en...
arXiv:2608.30609v1 Announce Type: cross Abstract: Large language models are increasingly capable in general, but their utility can remain modest in niche or understudied areas. One approach to addres...
ReliableRAG is a new framework for Retrieval-Augmented Generation that tackles misinformation in multi‑hop question answering. It extracts structured triples from retrieved documents, evaluates each triple’s reliability by combining semantic relevance to the query with credibility, and keeps only the top‑K reliable, non‑redundant triples. Using these refined triples, the system builds robust reasoning chains that filter out deceptive misinformation and produce accurate, trustworthy answers.
The study introduces a two‑dimensional framework to audit French news headlines, distinguishing salience framing—captured by four wording devices—from selection framing—captured by outlet‑level story form and high‑charge distributions. Using a 10,000‑headline supervision set annotated by LLMs and human arbitration, the authors classify 902,111 headlines from 25 outlets (2022‑2025) and find that salience and selection diverge yet correlate, that default thresholds inflate salience estimates, and that headlines mentioning Jews, the Far‑right, and Muslims exhibit the highest salience rates. The authors release their dataset, lexicons, and code, claiming it is the largest French headline framing audit to date.