OpenAI Blog
Oct 30, 2024
Introducing SimpleQA
A factuality benchmark called SimpleQA that measures the ability for language models to answer short, fact-seeking questions.
FACTS Benchmark Suite: Systematically evaluating the factuality of large language models
Systematically evaluating the factuality of large language models with the FACTS Benchmark Suite.
Related stories
arXiv AI
Aug 7
TriQua: Reconciling Granularity and Context in Factuality Evaluation
arXiv:2608. 05228v1 Announce Type: new Abstract: The "decompose-then-verify" paradigm for LLM factuality evaluation faces a fundamental trade-off: atomic facts, i.
Hugging Face Trending Papers
Jun 21
Not All Claims Are Equally Risky: FACTOR for Adaptive Verification in Factual Long-Form Generation
Large Language Models (LLMs) generate fluent long-form text, however, often add unsupported factual claims. Existing verification techniques improve factuality by grounding generation in external evidence.
arXiv AI
Jul 9
Comprehensive Evaluation of Large Language Model Responses: A Multi-Factor Scoring System
arXiv:2607. 06940v1 Announce Type: cross Abstract: The remarkable performance of large language models (LLMs) in linguistic tasks underscores an urgent need for comprehensive evaluation of their response quality.
arXiv Machine Learning
1d ago
L3Cube-IndicQuest v2: A Large-Scale Multilingual Benchmark for Evaluating Factual Knowledge of Large Language Models Across Indic Languages
arXiv:2608. 15535v1 Announce Type: cross Abstract: We present L3Cube-IndicQuest v2, a large-scale gold-standard multilingual question-answering benchmark for evaluating the India-specific factual knowledge of Large Language Models (LLMs).
arXiv AI
Jun 18
As Easy as Rocket Science: Assessing the Ability of Large Language Models to Interpret Negation in Figurative Language
arXiv:2606. 18922v1 Announce Type: cross Abstract: Figurative language and negation are two areas that challenge current language models, however, both are widely used throughout written and spoken language.
arXiv AI
Jun 3
Reasoning Structure of Large Language Models
arXiv:2606. 03883v1 Announce Type: new Abstract: Large reasoning models (LRMs) are often evaluated using metrics such as final-answer accuracy or token count.
arXiv AI
6d ago
Commonsense on Demand: Generating and Selectively Integrating Commonsense Knowledge for Natural Language Inference
arXiv:2507. 15100v3 Announce Type: replace-cross Abstract: Natural Language Inference (NLI) determines whether a premise entails, contradicts, or is neutral with respect to a hypothesis.
arXiv AI
Jun 29
Hybrid Fact-Checking that Integrates Knowledge Graphs, Large Language Models, and Search-Based Retrieval Agents Improves Interpretable Claim Verification
arXiv:2511. 03217v2 Announce Type: replace-cross Abstract: Large language models (LLMs) excel in generating fluent utterances but can lack reliable grounding in verified information.
arXiv AI
Aug 12
Is This Your Final Answer? Cross-Contextual Consistency as a Measure of LLM Credibility
arXiv:2608. 10315v1 Announce Type: cross Abstract: Large language models (LLMs) are powerful black-box systems, making it difficult to discern whether their answers reflect stable internal beliefs or superficial pattern matching.
Hugging Face Trending Papers
Jul 23
When Trivia Is Not Trivial: Everyday Knowledge Failures in Multilingual LLMs
Quiz rooms, trivia nights, and quiz shows challenge human knowledge across a wide range of topics, from canonical facts to everyday culture. In this paper, we examine whether large language models (LLMs) can perform competitively in such settings, using quiz-style questions to test them on both common and niche topics.
arXiv AI
Jul 8
Data Analysis in the Wild: Benchmarking Large Language Models Against Real-World Data Complexities
arXiv:2607. 06482v1 Announce Type: cross Abstract: Current benchmarks for evaluating Large Language Models (LLMs) in data analysis often fail to reflect real-world settings.