DeepMind Blog
Dec 9, 2025
FACTS Benchmark Suite: Systematically evaluating the factuality of large language models
Systematically evaluating the factuality of large language models with the FACTS Benchmark Suite.
Introducing SimpleQA
A factuality benchmark called SimpleQA that measures the ability for language models to answer short, fact-seeking questions.
Related stories
arXiv AI
Aug 7
TriQua: Reconciling Granularity and Context in Factuality Evaluation
arXiv:2608. 05228v1 Announce Type: new Abstract: The "decompose-then-verify" paradigm for LLM factuality evaluation faces a fundamental trade-off: atomic facts, i.
arXiv AI
Aug 11
Beyond Questions: Evaluating LLM's Knowledge Expression
arXiv:2605. 26937v2 Announce Type: replace-cross Abstract: Parametric knowledge in large language models (LLMs) is a cornerstone of their success, yet remains poorly understood.
arXiv AI
Jun 3
Reasoning Structure of Large Language Models
arXiv:2606. 03883v1 Announce Type: new Abstract: Large reasoning models (LRMs) are often evaluated using metrics such as final-answer accuracy or token count.
arXiv AI
Jun 10
LakeQA: An Exploratory QA Benchmark over a Million-Scale Data Lake
arXiv:2606. 10460v1 Announce Type: cross Abstract: Recent large language models (LLMs) have shown rapid progress in reading-based question answering (QA), where evidence is explicitly provided or can be trivially retrieved.
arXiv AI
Jul 1
SAGE: A Search-AuGmented Evaluation of Large Language Models on Free-Form QA
arXiv:2504. 07385v3 Announce Type: replace-cross Abstract: As Large Language Models (LLMs) become increasingly used for question-answering (QA), relying on static, pre-annotated references for evaluation poses significant challenges in cost, scalability, and completeness.
arXiv AI
Jun 26
NuclearQAv2: A Structured Benchmark for Evaluating Domain-Science Competence in Large Language Models
arXiv:2606. 27047v1 Announce Type: cross Abstract: Large language models (LLMs) have demonstrated strong performance across a wide range of tasks, but ensuring their reliability in highly technical domains remains a significant challenge.
arXiv AI
Jun 10
RankLLM: Weighted Ranking of LLMs by Quantifying Question Difficulty
arXiv:2602. 12424v2 Announce Type: replace-cross Abstract: Benchmarks establish a standardized evaluation framework to systematically assess the performance of large language models (LLMs), facilitating objective comparisons and driving advancements in the field.
OpenAI Blog
Dec 16, 2021
WebGPT: Improving the factual accuracy of language models through web browsing
We’ve fine-tuned GPT-3 to more accurately answer open-ended questions using a text-based web browser.
arXiv AI
6d ago
Towards Query-Agnostic RAG Evaluation via Query Coverage and Claim Verifiability
arXiv:2608. 11238v1 Announce Type: new Abstract: Retrieval-augmented generation improves the factuality of large language models by grounding responses in retrieved evidence, yet existing evaluation frameworks struggle to provide consistent, fine-grained diagnostics across the diverse spectrum of user queries, ranging from close-ended fact-seeking to open-ended explanatory requests.
Hugging Face Trending Papers
Jul 24
PatiGonit22K: A Comprehensive Dataset for Solving Complex Bengali MWPs
Mathematical Word Problems (MWPs) are an important benchmark for evaluating natural language understanding and quantitative reasoning. Despite recent progress in high resource languages, Bengali remains underexplored due to the limited availability of large scale annotated datasets.
Hugging Face Trending Papers
Jul 22
HalluTruthQA: A Fine-Grained Benchmark for Hallucination Detection, Localization, and Explanation in Arabic Question Answering
Large language models (LLMs) can generate fluent Arabic answers, yet factual errors remain difficult to detect, localize, explain, and verify. Existing hallucination benchmarks often provide response-level labels, with limited support for identifying the exact erroneous content, explaining why it is incorrect, or selecting the correct factual answer.