OpenAI Blog
TruthfulQA: Measuring how models mimic human falsehoods
Read the original on OpenAI Blog →The Flow has not summarised this story yet — read it at OpenAI Blog.
Related stories
arXiv AI
3d ago
Stress-Testing LLM Lie Detectors: Role-Play Failures and Spurious Correlations
The paper examines the reliability of lie detection probes for language models when the models adopt anti-factual personas, such as conspiracy theorists. A dataset of 8,916 human-reviewed responses from three LLMs was created, and eight existing probes were evaluated, revealing many fail to flag falsehoods under these personas. The authors also constructed confounder datasets showing that probes often track spurious correlations like instruction compliance, and propose a simple linear probe that performs best on both persona and confounder tests.
arXiv Machine Learning
5d ago
Fake News Theories: Harnessing Disciplinary Insights for Computational Modeling, Detection, and Explanation
The paper presents a theory-informed computational framework that converts cross-disciplinary theories of fake news into measurable features for automated detection and explanation. By reviewing theories from social sciences, psychology, economics, and more, the authors establish a broad theoretical foundation for computational modeling. Experiments on benchmark datasets demonstrate that theory-derived features are predictive, provide interpretable diagnostic signals, and that multi-feature models generally outperform individual features, though gains are modest.
arXiv Computation and Language
Sep 15
Can We Trust the Judges? Validation of Factuality Evaluation Methods via Answer Perturbation
arXiv:2609.15561v1 Announce Type: new Abstract: Evaluating the factual correctness of large language models (LLMs) is vital for many applications. But are our evaluation tools themselves trustworthy?...
arXiv AI
Jun 26
ConflictScore: Identifying and Measuring How Language Models Handle Conflicting Evidence
arXiv:2606. 26437v1 Announce Type: cross Abstract: Existing metrics for factuality and faithfulness evaluate whether an answer is supported or contradicted by its grounding documents, but they fail to capture when both supporting and contradicting evidence coexist.
arXiv AI
Jun 17
PseudoBench: Measuring How Agentic Auto-Research Fuels Pseudoscience
arXiv:2606. 18060v1 Announce Type: new Abstract: As Large Language Model based agents enter autonomous scientific research, their ability to resist pseudoscience becomes increasingly important.
arXiv AI
Aug 12
Graphical Models of False Information and Fact Checking Ecosystems
arXiv:2208. 11582v2 Announce Type: replace-cross Abstract: The wide spread of false information online, including misinformation and disinformation, has become a major problem for our highly digitised and globalised society.